Oct 30 20:19:06.691033 ip-10-0-130-195 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:19:07.073528 ip-10-0-130-195 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:07.073528 ip-10-0-130-195 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:19:07.073528 ip-10-0-130-195 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:07.073528 ip-10-0-130-195 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:07.079864 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079773    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.079864 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079858    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.079864 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079862    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.079864 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079865    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.079864 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079868    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.079864 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079871    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079875    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079880    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079884    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079888    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079892    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079895    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079898    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079901    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079903    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079906    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079908    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079911    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079913    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079916    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079918    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079921    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079924    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.080076 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079926    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079929    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079935    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079938    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079940    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079943    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079945    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079948    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079951    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079954    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079956    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079959    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079962    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079965    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079968    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079971    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079974    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079977    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079979    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.080539 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079982    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079984    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079987    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079989    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079991    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079994    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079996    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.079999    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080002    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080005    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080007    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080010    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080013    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080015    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080017    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080020    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080022    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080025    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080027    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.081010 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080030    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080032    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080035    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080038    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080041    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080043    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080046    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080049    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080052    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080054    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080057    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080059    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080062    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080064    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080067    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080069    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080072    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080074    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080077    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080079    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.081453 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080083    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080089    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080092    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080095    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080098    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080100    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080103    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080105    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080108    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080110    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080112    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080115    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080117    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080121    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080123    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080126    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080128    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080133    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.081943 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080136    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080138    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080141    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080143    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080146    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080919    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080926    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080929    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080932    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080935    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080938    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080941    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080944    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080947    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080950    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080953    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080955    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080958    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080960    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080963    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.082435 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080965    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080968    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080970    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080973    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080975    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080978    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080981    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080984    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080986    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080989    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080991    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080994    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080996    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.080999    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081001    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081004    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081008    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081011    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081014    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.083100 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081017    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081020    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081022    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081025    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081028    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081031    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081033    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081036    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081039    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081042    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081044    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081047    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081049    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081052    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081054    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081056    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081059    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081061    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081064    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.083557 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081066    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081069    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081071    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081074    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081076    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081079    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081081    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081084    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081086    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081089    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081091    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081094    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081096    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081099    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081102    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081104    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081107    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081110    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081113    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.084040 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081115    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081118    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081120    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081123    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081126    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081129    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081131    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081133    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081136    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081139    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081141    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081144    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081147    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081150    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081152    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081156    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081160    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081163    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081166    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.084514 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081170    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081172    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081175    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081178    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081180    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081183    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081186    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081188    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081192    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081194    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081197    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081200    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081202    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081205    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081207    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081755    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081763    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081766    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081770    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081772    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.084986 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081775    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081778    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081780    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081783    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081786    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081788    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081791    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081794    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081797    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081799    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081802    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081804    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081807    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081809    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081812    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081815    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081817    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081820    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081823    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081826    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.085458 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081828    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081831    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081833    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081851    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081855    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081858    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081861    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081864    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081867    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081870    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081873    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081876    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081878    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081881    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081883    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081886    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081888    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081890    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081894    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.085941 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081896    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081899    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081902    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081904    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081906    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081909    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081911    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081914    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081916    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081919    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081921    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081924    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081926    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081929    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081931    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081934    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081937    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081939    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081943    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.086403 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081947    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081950    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081953    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081957    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081959    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081962    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081965    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081968    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081970    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081973    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081975    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081977    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081980    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081983    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081985    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081988    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081990    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081993    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081995    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.086878 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.081998    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082001    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082003    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082006    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082010    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082014    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082018    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082021    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082024    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082026    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082029    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082031    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082033    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082036    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082038    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082041    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082044    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082047    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082049    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082052    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.087340 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082055    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082057    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082060    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082062    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082167    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082178    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082189    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082195    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082202    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082206    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082210    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082214    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082217    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082221    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082224    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082227    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082233    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082236    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082239    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082242    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082245    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082248    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082251    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:19:07.087826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082254    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082257    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082260    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082263    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082267    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082270    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082273    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082276    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082280    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082284    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082295    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082299    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082301    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082305    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082309    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082312    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082314    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082317    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082320    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082325    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082328    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082333    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082338    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082343    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:19:07.088426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082349    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082354    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082357    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082360    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082363    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082366    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082368    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082371    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082374    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082377    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082379    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082384    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082387    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082389    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082393    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082396    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082399    2577 flags.go:64] FLAG: --help="false"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082401    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082405    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082408    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082411    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082414    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082418    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:19:07.089040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082420    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082423    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082426    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082429    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082432    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082435    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082438    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082441    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082444    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082447    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082451    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082454    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082456    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082460    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082463    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082468    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082471    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082474    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082477    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082479    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082483    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082485    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082488    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082494    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082497    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:19:07.089576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082500    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082503    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082506    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082510    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082512    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082515    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082518    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082521    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082528    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082531    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082534    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082537    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082540    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082543    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082546    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082549    2577 flags.go:64] FLAG: --port="10250"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082552    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082557    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-07de1ee85cd06b297"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082560    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082563    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082566    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082568    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082572    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082575    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082578    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082581    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:19:07.090202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082584    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082587    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082590    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082593    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082598    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082601    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082604    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082607    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082610    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082612    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082616    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082620    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082623    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082626    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082629    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082633    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082635    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082638    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082641    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082644    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082649    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082652    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082655    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082660    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082663    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:19:07.090843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082665    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082668    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082671    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082674    2577 flags.go:64] FLAG: --v="2"
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082678    2577 flags.go:64] FLAG: --version="false"
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082683    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082687    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.082690    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082847    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082851    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082854    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082858    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082861    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082864    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082867    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082869    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082872    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082874    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082879    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082881    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082884    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.091421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082887    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082889    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082892    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082895    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082899    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082902    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082905    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082908    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082911    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082914    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082917    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082919    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082922    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082925    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082927    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082930    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082932    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082935    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082937    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.091983 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082940    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082943    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082945    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082949    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082952    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082954    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082957    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082960    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082962    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082965    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082967    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082970    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082973    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082976    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082978    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082981    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082983    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082986    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082988    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082991    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.092439 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082993    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.082996    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083000    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083002    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083005    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083007    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083010    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083012    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083015    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083018    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083020    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083024    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083028    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083031    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083034    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083038    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083041    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083043    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083046    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083048    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.092947 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083051    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083053    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083056    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083058    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083061    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083064    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083067    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083069    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083072    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083074    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083077    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083080    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083082    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083085    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083088    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083091    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083096    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083098    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083101    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.093423 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083104    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083106    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083109    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083112    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083114    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083117    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083120    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083123    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083126    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083129    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083131    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083133    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083136    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083138    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.083168    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083208    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.093925 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083212    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083216    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083219    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083222    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083225    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083228    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083230    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083233    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083236    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083238    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083242    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083245    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083247    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083250    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083252    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083255    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083257    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083259    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083262    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083264    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.094317 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083267    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083269    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083272    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083274    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083278    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083280    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083283    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083286    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083288    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083291    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083293    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083296    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083299    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083301    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083307    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083309    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083312    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083314    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083317    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.094813 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083319    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083322    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083325    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083329    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083331    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083335    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083339    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083342    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083345    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083348    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083351    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083354    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083357    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083360    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083363    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083365    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083368    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083373    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083376    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.095254 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083379    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083382    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083385    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083387    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083390    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083392    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083395    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083397    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083400    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083403    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083405    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083408    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083410    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083413    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083415    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083418    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083422    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083425    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083427    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083430    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.095702 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083432    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083435    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083437    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083440    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083443    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083445    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083447    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083450    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083452    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083455    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083459    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083462    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083466    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083469    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083472    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083475    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083477    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083480    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083482    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083484    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.096271 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083487    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083489    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083493    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083495    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083497    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083500    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.083502    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.090085    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.090099    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.090803    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.090810    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090870    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090876    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090879    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090881    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090885    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090887    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090890    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090893    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090896    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.096751 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090901    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090904    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090906    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090909    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090911    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090914    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090916    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090919    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090921    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090924    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090926    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090929    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090932    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090935    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090938    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090941    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090943    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090946    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.097282 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090949    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090951    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090957    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090959    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090961    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090964    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090966    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090969    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090971    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090974    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090976    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090979    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090981    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090984    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090986    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090988    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090991    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090993    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.097719 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.090998    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091001    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091005    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091008    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091011    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091013    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091016    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091019    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091021    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091025    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091027    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091030    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091032    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091035    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091038    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091040    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091043    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091045    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091047    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.098181 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091050    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091052    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091055    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091057    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091059    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091062    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091064    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091067    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091069    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091071    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091074    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091076    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091079    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091081    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091084    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091086    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091088    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091095    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091098    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091100    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.098642 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091103    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091105    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091109    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091113    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091116    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091119    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091122    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091124    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091127    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091129    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091131    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091134    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091136    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091138    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091141    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091143    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091145    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091148    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091150    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.099133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091153    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.091181    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091315    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091319    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091322    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091325    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091327    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091329    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091332    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091334    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091337    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091339    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091342    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091344    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091347    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.099607 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091349    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091352    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091355    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091358    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091360    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091363    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091365    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091367    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091370    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091372    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091374    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091377    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091379    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091382    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091384    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091387    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091389    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091391    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091394    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.100170 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091396    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091399    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091401    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091403    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091406    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091408    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091411    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091413    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091415    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091418    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091420    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091423    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091425    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091428    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091431    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091433    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091436    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091439    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091441    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091444    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.100692 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091446    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091448    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091451    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091453    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091456    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091458    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091460    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091463    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091465    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091467    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091470    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091472    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091475    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091477    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091480    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091482    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091484    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091488    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091492    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.101229 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091494    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091498    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091500    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091503    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091505    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091508    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091510    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091512    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091515    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091518    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091523    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091526    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091528    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091530    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091533    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091535    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091537    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091540    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091542    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091544    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.101685 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091547    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091549    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091552    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091554    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091556    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091559    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091561    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091564    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091566    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091568    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091571    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091573    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091577    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091580    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.091605    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:07.102190 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091638    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091643    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091646    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091649    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091651    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091654    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091656    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091659    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091662    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091664    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091666    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091669    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091671    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091674    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091676    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091679    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091681    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091683    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091686    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:07.102595 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091688    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091691    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091693    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091695    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091698    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091700    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091703    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091705    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091707    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091710    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091712    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091715    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091717    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091720    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091722    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091725    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091727    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091730    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091732    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091734    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:07.103207 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091751    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091754    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091757    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091759    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091762    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091765    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091767    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091769    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091772    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091774    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091777    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091779    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091782    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091784    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091786    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091789    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091791    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091794    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091796    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091799    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:07.103673 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091801    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091803    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091806    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091809    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091813    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091816    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091818    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091821    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091823    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091827    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091831    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091834    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091837    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091841    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091844    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091847    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091849    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091851    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091854    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091856    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:07.104163 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091859    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091862    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091864    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091867    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091869    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091871    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091874    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091876    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091879    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091881    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091883    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091886    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091888    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091891    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091893    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091896    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091900    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091902    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091905    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091907    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:07.104668 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091910    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091912    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091914    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091919    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091923    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091926    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:07.091928    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.091975    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.093993    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.094962    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.095059    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.095107    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.096850    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:07.105151 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.102587    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:07.115033 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.115012    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:19:07.118843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.118827    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 20:19:07.119643 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.119618    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:19:07.124211 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.124186    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 91d4c5d2-ce63-4a60-8963-d45a438a0f04:/dev/nvme0n1p3 d9219cde-95c5-49f7-8807-55e9791b21e3:/dev/nvme0n1p4]
Oct 30 20:19:07.124262 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.124212    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:19:07.129367 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.129256    2577 manager.go:217] Machine: {Timestamp:2026-10-30 20:19:07.127879633 +0000 UTC m=+0.325310662 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100331 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24e34c235d2b0180964f5c6f2916d1 SystemUUID:ec24e34c-235d-2b01-8096-4f5c6f2916d1 BootID:9484d53e-c2bc-4da1-945a-96140e4812b4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:a5:df:78:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:a5:df:78:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:11:9c:e6:71:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:19:07.129367 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.129366    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:19:07.129491 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.129480    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:19:07.132058 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.132031    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:19:07.132190 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.132059    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-195.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:19:07.132715 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.132705    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:19:07.132766 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.132717    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:19:07.132766 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.132753    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:07.132824 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.132767    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:07.133512 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.133502    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:19:07.136230 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.136217    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:19:07.136339 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.136330    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:19:07.137701 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.137690    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:19:07.137754 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.137704    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:19:07.137754 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.137714    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:19:07.137754 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.137724    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:19:07.137754 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.137731    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:19:07.138100 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.138089    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:07.140465 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.140448    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:19:07.144984 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.144970    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:07.145680 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145668    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:19:07.145712 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145698    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:19:07.145883 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145873    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:19:07.145913 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145889    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:19:07.145913 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145895    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:19:07.145913 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145901    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:19:07.145913 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145906    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:19:07.145913 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.145913    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:19:07.146382 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146373    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:19:07.146412 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146383    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:19:07.146412 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146390    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:19:07.146412 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146396    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:19:07.146412 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146404    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:19:07.146532 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146414    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:19:07.146936 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146926    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:19:07.146936 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.146936    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:19:07.150586 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.150573    2577 server.go:1263] "Started kubelet"
Oct 30 20:19:07.150663 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.150640    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:19:07.153680 ip-10-0-130-195 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:19:07.154202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.153175    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:19:07.154202 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.153966    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:19:07.156335 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.156317    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:19:07.157711 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.157690    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:19:07.159532 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.159509    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:19:07.161947 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.161922    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:19:07.162284 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.162265    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:19:07.162382 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.162366    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-195.us-west-1.compute.internal\" not found"
Oct 30 20:19:07.162850 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.162810    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:19:07.162850 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.162830    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:19:07.163003 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.162809    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:19:07.163080 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.162947    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:19:07.163080 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163080    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:19:07.163241 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163225    2577 factory.go:55] Registering systemd factory
Oct 30 20:19:07.163292 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163246    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:19:07.163461 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163450    2577 factory.go:153] Registering CRI-O factory
Oct 30 20:19:07.163528 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163463    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:19:07.163528 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163501    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:19:07.163528 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163519    2577 factory.go:103] Registering Raw factory
Oct 30 20:19:07.163528 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.163529    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:19:07.164383 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.164371    2577 manager.go:319] Starting recovery of all containers
Oct 30 20:19:07.173725 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.173704    2577 manager.go:324] Recovery completed
Oct 30 20:19:07.178577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.178563    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.180571 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.180553    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.180636 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.180581    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.180636 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.180594    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.181165 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.181150    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:19:07.181165 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.181164    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:19:07.181269 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.181182    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:19:07.181269 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.181155    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.182362 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.182347    2577 policy_none.go:50] "Start"
Oct 30 20:19:07.182362 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.182364    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:19:07.182501 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.182374    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:19:07.182501 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.182348    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.182501 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.182455    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.182501 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.182475    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.183440 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.183430    2577 policy_none.go:44] "Start"
Oct 30 20:19:07.214558 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.214436    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:19:07.219075 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219062    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:19:07.219162 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.219092    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:19:07.219162 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219100    2577 server.go:87] "Starting device plugin registration server"
Oct 30 20:19:07.219325 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219312    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:19:07.219374 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219329    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:19:07.219433 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219415    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:19:07.219512 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219500    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:19:07.219566 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.219513    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:19:07.219944 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.219925    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:19:07.220003 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.219966    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-195.us-west-1.compute.internal\" not found"
Oct 30 20:19:07.295702 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.295676    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-195.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:19:07.296944 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.295677    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-195.us-west-1.compute.internal.18e3681c3f18a641  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:19:07.150542401 +0000 UTC m=+0.347973429,LastTimestamp:2026-10-30 20:19:07.150542401 +0000 UTC m=+0.347973429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.us-west-1.compute.internal,}"
Oct 30 20:19:07.298444 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.298421    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:19:07.299786 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.299770    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:19:07.299855 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.299790    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:19:07.299855 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.299806    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:19:07.299855 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.299833    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:19:07.319772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.319753    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.320475 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.320462    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.320533 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.320493    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.320533 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.320509    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.320594 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.320536    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.345212 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.345157    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-195.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:19:07.371494 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.371473    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:19:07.371569 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.371522    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k2wpz"
Oct 30 20:19:07.400558 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.400533    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"]
Oct 30 20:19:07.400621 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.400592    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.401447 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.401433    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.401490 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.401461    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.401490 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.401475    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.402975 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.402962    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.403126 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.403166 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403142    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.403643 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403627    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.403710 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403634    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.403710 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403656    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.403710 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403666    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.403710 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403676    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.403710 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.403691    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.405139 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.405113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.405233 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.405143    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:07.405862 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.405848    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:07.405961 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.405871    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:07.405961 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.405881    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:07.437813 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.437795    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-195.us-west-1.compute.internal\" not found" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.440232 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.440217    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-195.us-west-1.compute.internal\" not found" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.453277 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.453261    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:07.454884 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.454870    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:07.465725 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.465706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.465813 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.465730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.465813 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.465766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/066c09478c1923ae07faa25b9022b5b3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"066c09478c1923ae07faa25b9022b5b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.551683 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.551663    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:07.552276 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.552262    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:07.553508 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.553492    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.553557 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:07.553516    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.us-west-1.compute.internal\": node \"ip-10-0-130-195.us-west-1.compute.internal\" not found"
Oct 30 20:19:07.563287 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.563269    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.566452 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.566437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.566492 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.566461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.566492 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.566478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/066c09478c1923ae07faa25b9022b5b3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"066c09478c1923ae07faa25b9022b5b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.566554 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.566540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/066c09478c1923ae07faa25b9022b5b3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"066c09478c1923ae07faa25b9022b5b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.566588 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.566555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.566588 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.566541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.655104 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.655051    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:07.655790 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.655777    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.714815 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.714795    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:07.740979 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.740954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:07.743104 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:07.743088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:19:08.095170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.095099    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:19:08.095662 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.095207    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:19:08.095662 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.095237    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:19:08.095662 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.095253    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:19:08.095662 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.095311    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab4a4e8a057d64e13b2573a04657ce20-dcfeed6681acbb51.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-195.us-west-1.compute.internal.18e3681c40e34669\": read tcp 10.0.130.195:32838->77.112.75.128:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-195.us-west-1.compute.internal.18e3681c40e34669  default   5239 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-195.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:19:07 +0000 UTC,LastTimestamp:2026-10-30 20:19:07.320515507 +0000 UTC m=+0.517946535,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.us-west-1.compute.internal,}"
Oct 30 20:19:08.138115 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.138100    2577 apiserver.go:52] "Watching apiserver"
Oct 30 20:19:08.162459 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.162439    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:19:08.375478 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.375397    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:14:07 +0000 UTC" deadline="2028-06-24 22:09:32.569914809 +0000 UTC"
Oct 30 20:19:08.375478 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.375434    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14473h50m24.194483496s"
Oct 30 20:19:08.404122 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.404104    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:19:08.404463 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.404443    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-jw4mv","openshift-dns/node-resolver-blv5n","kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal","openshift-multus/multus-cfstj","openshift-multus/network-metrics-daemon-8ss9g","kube-system/konnectivity-agent-pxcqk","openshift-image-registry/node-ca-qtnkw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-dgvmc","openshift-ovn-kubernetes/ovnkube-node-7pz42","kube-system/global-pull-secret-syncer-f84bz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p","openshift-multus/multus-additional-cni-plugins-rk542","openshift-network-diagnostics/network-check-target-tj2kt"]
Oct 30 20:19:08.406520 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.406503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.408227 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.408213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.409921 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.409906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.411781 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.411768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:08.411846 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.411820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:08.413563 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.413547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.415465 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.415450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.417393 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.417379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.419422 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.419401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.421406 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.421389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.421487 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.421437    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:08.423341 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.423327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.425468 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.425456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.427516 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.427501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:08.427564 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.427551    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462323    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwff\""
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462450    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462451    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462461    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k74qc\""
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462446    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462450    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.462591 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462563    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6wxmv\""
Oct 30 20:19:08.463066 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.462877    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:19:08.463066 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.463000    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:19:08.463235 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.463136    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.463293 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.463242    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.463796 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.463782    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:19:08.465860 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.465844    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:19:08.465938 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.465845    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:19:08.466163 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.466151    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vn2fz\""
Oct 30 20:19:08.468725 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.468713    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.469191 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.469174    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kv88f\""
Oct 30 20:19:08.469274 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.469194    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:19:08.469358 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.469321    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.469445 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.469430    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.472381 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-systemd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.472381 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-systemd\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.472521 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-var-lib-kubelet\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.472521 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/707f544c-cf0c-4c54-bd0f-6dec8791c59f-host\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.472521 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-cnibin\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.472521 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-systemd-units\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-netns\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-sys-fs\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkff\" (UniqueName: \"kubernetes.io/projected/e97088a0-b646-48d9-818b-c932841333e4-kube-api-access-jkkff\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-k8s-cni-cncf-io\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.472789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-multus\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-kubelet\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-ovn\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-modprobe-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-sys\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkwp\" (UniqueName: \"kubernetes.io/projected/707f544c-cf0c-4c54-bd0f-6dec8791c59f-kube-api-access-wqkwp\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.472997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-kubernetes\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.473040 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-netd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/707f544c-cf0c-4c54-bd0f-6dec8791c59f-serviceca\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-socket-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-device-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-script-lib\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-socket-dir-parent\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc44z\" (UniqueName: \"kubernetes.io/projected/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-kube-api-access-nc44z\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-node-log\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-tuned\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-multus-certs\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-dbus\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-netns\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-log-socket\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-conf\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-host\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c546q\" (UniqueName: \"kubernetes.io/projected/2f19916b-62f3-4a81-ab47-4cde132463c5-kube-api-access-c546q\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-bin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-hostroot\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213915b5-832d-44c7-a046-12833d574630-host-slash\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-etc-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.473964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213915b5-832d-44c7-a046-12833d574630-iptables-alerter-script\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-system-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkkh\" (UniqueName: \"kubernetes.io/projected/cb0e115f-349e-4f69-b643-336b58e3c56c-kube-api-access-gvkkh\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-var-lib-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysconfig\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-run\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.473988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cnibin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-etc-kubernetes\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-slash\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-config\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vsd\" (UniqueName: \"kubernetes.io/projected/c9789942-84f9-4af9-ba2d-52d3115b5882-kube-api-access-89vsd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-lib-modules\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-registration-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bac513-58af-4a06-a6e6-f419b8a9bc60-tmp-dir\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-daemon-config\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-tmp\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.474633 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-env-overrides\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4bac513-58af-4a06-a6e6-f419b8a9bc60-hosts-file\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cni-binary-copy\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-conf-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrsz\" (UniqueName: \"kubernetes.io/projected/213915b5-832d-44c7-a046-12833d574630-kube-api-access-ncrsz\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxg45\" (UniqueName: \"kubernetes.io/projected/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-kube-api-access-kxg45\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-system-cni-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjflc\" (UniqueName: \"kubernetes.io/projected/b4bac513-58af-4a06-a6e6-f419b8a9bc60-kube-api-access-hjflc\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bded661-99cb-4a66-a36b-4e0a446321ef-konnectivity-ca\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-kubelet\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-bin\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9789942-84f9-4af9-ba2d-52d3115b5882-ovn-node-metrics-cert\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-kubelet-config\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bded661-99cb-4a66-a36b-4e0a446321ef-agent-certs\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.475242 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.474691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.507434 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.507416    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:19:08.507691 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.507677    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lm5vf\""
Oct 30 20:19:08.507773 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.507722    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5x4ch\""
Oct 30 20:19:08.510835 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.510813    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwdx\""
Oct 30 20:19:08.557187 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.557162    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:19:08.557265 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.557214    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtwsd"
Oct 30 20:19:08.560377 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560358    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:19:08.560511 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560362    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:19:08.560511 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560398    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.560511 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560471    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbmn9\""
Oct 30 20:19:08.560669 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560620    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:19:08.560669 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560620    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.560669 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560649    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.560669 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560656    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:19:08.560827 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560667    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:08.560827 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560679    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:19:08.560827 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560689    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:19:08.560908 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560839    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:19:08.560937 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.560912    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:19:08.575045 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-multus-certs\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.575127 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-dbus\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.575127 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.575127 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-netns\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.575127 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575117    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-multus-certs\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-log-socket\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-netns\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-conf\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-log-socket\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575201    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-dbus\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-host\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c546q\" (UniqueName: \"kubernetes.io/projected/2f19916b-62f3-4a81-ab47-4cde132463c5-kube-api-access-c546q\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-host\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-bin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575295    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-conf\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-hostroot\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.575315 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575311    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-bin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213915b5-832d-44c7-a046-12833d574630-host-slash\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575340    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-hostroot\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-etc-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213915b5-832d-44c7-a046-12833d574630-host-slash\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213915b5-832d-44c7-a046-12833d574630-iptables-alerter-script\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-system-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575423    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-etc-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkkh\" (UniqueName: \"kubernetes.io/projected/cb0e115f-349e-4f69-b643-336b58e3c56c-kube-api-access-gvkkh\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-var-lib-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575460    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-system-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysconfig\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-run\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575498    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-var-lib-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cnibin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575522    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysconfig\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-run\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-etc-kubernetes\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576064 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575548    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cnibin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575572    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-etc-kubernetes\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-slash\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-config\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575643    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-slash\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.575909    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213915b5-832d-44c7-a046-12833d574630-iptables-alerter-script\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576153    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-config\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vsd\" (UniqueName: \"kubernetes.io/projected/c9789942-84f9-4af9-ba2d-52d3115b5882-kube-api-access-89vsd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-lib-modules\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-registration-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bac513-58af-4a06-a6e6-f419b8a9bc60-tmp-dir\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576373    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-registration-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-daemon-config\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-lib-modules\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-tmp\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.576923 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-env-overrides\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4bac513-58af-4a06-a6e6-f419b8a9bc60-hosts-file\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576630    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bac513-58af-4a06-a6e6-f419b8a9bc60-tmp-dir\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cni-binary-copy\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4bac513-58af-4a06-a6e6-f419b8a9bc60-hosts-file\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-conf-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrsz\" (UniqueName: \"kubernetes.io/projected/213915b5-832d-44c7-a046-12833d574630-kube-api-access-ncrsz\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576796    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-conf-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxg45\" (UniqueName: \"kubernetes.io/projected/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-kube-api-access-kxg45\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576842    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-env-overrides\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-system-cni-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577021    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-system-cni-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.576992    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-daemon-config\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjflc\" (UniqueName: \"kubernetes.io/projected/b4bac513-58af-4a06-a6e6-f419b8a9bc60-kube-api-access-hjflc\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bded661-99cb-4a66-a36b-4e0a446321ef-konnectivity-ca\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-kubelet\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-bin\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.577540 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-kubelet\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9789942-84f9-4af9-ba2d-52d3115b5882-ovn-node-metrics-cert\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577282    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-bin\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-kubelet-config\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bded661-99cb-4a66-a36b-4e0a446321ef-agent-certs\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-systemd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577426    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-kubelet-config\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-systemd\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-var-lib-kubelet\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-systemd\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577499    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/707f544c-cf0c-4c54-bd0f-6dec8791c59f-host\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/707f544c-cf0c-4c54-bd0f-6dec8791c59f-host\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-cnibin\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-systemd-units\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.578316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-netns\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-sys-fs\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkff\" (UniqueName: \"kubernetes.io/projected/e97088a0-b646-48d9-818b-c932841333e4-kube-api-access-jkkff\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cni-binary-copy\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-k8s-cni-cncf-io\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-k8s-cni-cncf-io\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-multus\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577731    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-var-lib-kubelet\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-cnibin\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-kubelet\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-ovn\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-modprobe-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-sys\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkwp\" (UniqueName: \"kubernetes.io/projected/707f544c-cf0c-4c54-bd0f-6dec8791c59f-kube-api-access-wqkwp\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-kubernetes\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.579917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.577996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-netd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/707f544c-cf0c-4c54-bd0f-6dec8791c59f-serviceca\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-socket-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-device-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-script-lib\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-socket-dir-parent\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc44z\" (UniqueName: \"kubernetes.io/projected/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-kube-api-access-nc44z\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bded661-99cb-4a66-a36b-4e0a446321ef-konnectivity-ca\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-node-log\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-systemd-units\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578431    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578456    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-netns\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.582509 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-tuned\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578503    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-systemd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-multus\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-kubelet\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578655    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-sys\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.578838    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.578922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-socket-dir-parent\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.579060    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.579122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:19:09.079090016 +0000 UTC m=+2.276521035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579139    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-device-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579155    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-socket-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-ovn\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-kubernetes\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579359    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.583976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-node-log\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579443    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579495    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-netd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.580314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:09.079223292 +0000 UTC m=+2.276654309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.580334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-modprobe-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.579431    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-sys-fs\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.580595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.580654    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/707f544c-cf0c-4c54-bd0f-6dec8791c59f-serviceca\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.581468    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.581662    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-script-lib\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.581707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9789942-84f9-4af9-ba2d-52d3115b5882-ovn-node-metrics-cert\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.581754    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bded661-99cb-4a66-a36b-4e0a446321ef-agent-certs\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.584582 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.584108    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-tmp\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.584982 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.584876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-tuned\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.638595 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.638523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c546q\" (UniqueName: \"kubernetes.io/projected/2f19916b-62f3-4a81-ab47-4cde132463c5-kube-api-access-c546q\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.639467 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.639444    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkkh\" (UniqueName: \"kubernetes.io/projected/cb0e115f-349e-4f69-b643-336b58e3c56c-kube-api-access-gvkkh\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:08.639787 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.639765    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vsd\" (UniqueName: \"kubernetes.io/projected/c9789942-84f9-4af9-ba2d-52d3115b5882-kube-api-access-89vsd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.640021 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.639991    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:08.640021 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.640016    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:08.640187 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.640030    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:08.640187 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:08.640111    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:09.140092461 +0000 UTC m=+2.337523480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:08.640773 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.640728    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjflc\" (UniqueName: \"kubernetes.io/projected/b4bac513-58af-4a06-a6e6-f419b8a9bc60-kube-api-access-hjflc\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.641396 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.641182    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrsz\" (UniqueName: \"kubernetes.io/projected/213915b5-832d-44c7-a046-12833d574630-kube-api-access-ncrsz\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.641503 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.641482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxg45\" (UniqueName: \"kubernetes.io/projected/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-kube-api-access-kxg45\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.641773 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.641757    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkff\" (UniqueName: \"kubernetes.io/projected/e97088a0-b646-48d9-818b-c932841333e4-kube-api-access-jkkff\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.642278 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.642264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkwp\" (UniqueName: \"kubernetes.io/projected/707f544c-cf0c-4c54-bd0f-6dec8791c59f-kube-api-access-wqkwp\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.643104 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.643088    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc44z\" (UniqueName: \"kubernetes.io/projected/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-kube-api-access-nc44z\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.735419 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.735375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:19:08.759522 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.759498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:19:08.775019 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.774997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cfstj"
Oct 30 20:19:08.791215 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.791194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:08.811339 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.811318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:19:08.827228 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.827210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:19:08.828881 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.828858    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066c09478c1923ae07faa25b9022b5b3.slice/crio-fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55 WatchSource:0}: Error finding container fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55: Status 404 returned error can't find the container with id fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55
Oct 30 20:19:08.829324 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.829309    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707f544c_cf0c_4c54_bd0f_6dec8791c59f.slice/crio-5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e WatchSource:0}: Error finding container 5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e: Status 404 returned error can't find the container with id 5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e
Oct 30 20:19:08.829886 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.829866    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5151f952d069614d161163f3dcf066f0.slice/crio-9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8 WatchSource:0}: Error finding container 9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8: Status 404 returned error can't find the container with id 9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8
Oct 30 20:19:08.833763 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.833735    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:19:08.836978 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.836960    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213915b5_832d_44c7_a046_12833d574630.slice/crio-461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623 WatchSource:0}: Error finding container 461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623: Status 404 returned error can't find the container with id 461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623
Oct 30 20:19:08.855068 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.855048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:08.863324 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.863306    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9789942_84f9_4af9_ba2d_52d3115b5882.slice/crio-b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78 WatchSource:0}: Error finding container b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78: Status 404 returned error can't find the container with id b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78
Oct 30 20:19:08.864481 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.864463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:19:08.872494 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.872477    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97088a0_b646_48d9_818b_c932841333e4.slice/crio-9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8 WatchSource:0}: Error finding container 9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8: Status 404 returned error can't find the container with id 9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8
Oct 30 20:19:08.880525 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:08.880510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:19:08.889564 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:08.889525    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f19916b_62f3_4a81_ab47_4cde132463c5.slice/crio-efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3 WatchSource:0}: Error finding container efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3: Status 404 returned error can't find the container with id efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3
Oct 30 20:19:09.082043 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.082021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:09.082154 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.082051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:09.082154 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.082136    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:09.082236 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.082172    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:09.082236 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.082181    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:10.082167718 +0000 UTC m=+3.279598737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:09.082236 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.082229    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:19:10.082216856 +0000 UTC m=+3.279647874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:09.182929 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.182860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:09.183508 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.183016    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:09.183508 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.183037    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:09.183508 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.183048    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:09.183508 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:09.183099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:10.183085041 +0000 UTC m=+3.380516077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:09.304369 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.304318    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p" event={"ID":"e97088a0-b646-48d9-818b-c932841333e4","Type":"ContainerStarted","Data":"9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8"}
Oct 30 20:19:09.305439 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.305417    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78"}
Oct 30 20:19:09.306583 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.306564    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dgvmc" event={"ID":"213915b5-832d-44c7-a046-12833d574630","Type":"ContainerStarted","Data":"461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623"}
Oct 30 20:19:09.307660 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.307641    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"5151f952d069614d161163f3dcf066f0","Type":"ContainerStarted","Data":"9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8"}
Oct 30 20:19:09.308683 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.308664    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qtnkw" event={"ID":"707f544c-cf0c-4c54-bd0f-6dec8791c59f","Type":"ContainerStarted","Data":"5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e"}
Oct 30 20:19:09.309589 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.309573    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"066c09478c1923ae07faa25b9022b5b3","Type":"ContainerStarted","Data":"fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55"}
Oct 30 20:19:09.310565 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.310544    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerStarted","Data":"efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3"}
Oct 30 20:19:09.540737 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.540659    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:09.559789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.559717    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:14:08 +0000 UTC" deadline="2028-07-19 23:38:19.391596382 +0000 UTC"
Oct 30 20:19:09.559789 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.559780    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15075h19m9.831820865s"
Oct 30 20:19:09.591506 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.591483    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:09.616520 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:09.616495    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bac513_58af_4a06_a6e6_f419b8a9bc60.slice/crio-9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb WatchSource:0}: Error finding container 9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb: Status 404 returned error can't find the container with id 9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb
Oct 30 20:19:09.686386 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:09.686359    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fec4f0_9aa9_4e1f_be2e_66c8c4968e97.slice/crio-beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1 WatchSource:0}: Error finding container beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1: Status 404 returned error can't find the container with id beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1
Oct 30 20:19:09.689538 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:09.689519    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:09.699299 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:09.699279    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bded661_99cb_4a66_a36b_4e0a446321ef.slice/crio-8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822 WatchSource:0}: Error finding container 8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822: Status 404 returned error can't find the container with id 8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822
Oct 30 20:19:09.719800 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:09.719777    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f681324_576a_4ae5_95fd_2a7e8f8ce67a.slice/crio-e3c405956a464aa785180a3539584d2b88582fbc5cd95200eb2adfe1dd22b29d WatchSource:0}: Error finding container e3c405956a464aa785180a3539584d2b88582fbc5cd95200eb2adfe1dd22b29d: Status 404 returned error can't find the container with id e3c405956a464aa785180a3539584d2b88582fbc5cd95200eb2adfe1dd22b29d
Oct 30 20:19:10.090372 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.089937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:10.090372 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.089995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:10.090372 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.090158    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:10.090372 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.090222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:12.090203008 +0000 UTC m=+5.287634027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:10.090372 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.090302    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:10.090372 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.090338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:19:12.090326699 +0000 UTC m=+5.287757735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:10.191479 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.191189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:10.191479 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.191348    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:10.191479 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.191369    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:10.191479 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.191382    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:10.191479 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.191438    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:12.191419637 +0000 UTC m=+5.388850693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:10.300964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.300473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:10.300964 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.300588    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:10.300964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.300674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:10.300964 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.300750    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:10.300964 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.300834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:10.300964 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:10.300918    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:10.319497 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.319461    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jw4mv" event={"ID":"1f681324-576a-4ae5-95fd-2a7e8f8ce67a","Type":"ContainerStarted","Data":"e3c405956a464aa785180a3539584d2b88582fbc5cd95200eb2adfe1dd22b29d"}
Oct 30 20:19:10.321068 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.321038    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pxcqk" event={"ID":"4bded661-99cb-4a66-a36b-4e0a446321ef","Type":"ContainerStarted","Data":"8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822"}
Oct 30 20:19:10.323330 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.323303    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cfstj" event={"ID":"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97","Type":"ContainerStarted","Data":"beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1"}
Oct 30 20:19:10.325995 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.325969    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-blv5n" event={"ID":"b4bac513-58af-4a06-a6e6-f419b8a9bc60","Type":"ContainerStarted","Data":"9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb"}
Oct 30 20:19:10.560359 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.560230    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:14:08 +0000 UTC" deadline="2028-08-01 08:35:25.013949894 +0000 UTC"
Oct 30 20:19:10.560359 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:10.560268    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15372h16m14.453686507s"
Oct 30 20:19:12.113574 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:12.113530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:12.114299 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:12.113590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:12.114299 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.113742    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:12.114299 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.113828    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:16.113808684 +0000 UTC m=+9.311239707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:12.114299 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.114260    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:12.114565 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.114314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:19:16.114296724 +0000 UTC m=+9.311727748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:12.214866 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:12.214654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:12.214866 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.214858    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:12.214866 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.214877    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:12.215181 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.214890    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:12.215181 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.214943    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:16.214925646 +0000 UTC m=+9.412356663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:12.300727 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:12.300655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:12.300925 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.300802    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:12.301356 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:12.301188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:12.301356 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.301301    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:12.301543 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:12.301363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:12.301543 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:12.301432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:13.360345 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:13.360071    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="d0342510f3dcdad0c8a46ef504b0e8f8084e154187d8a6d021ced38ad08db77c" exitCode=0
Oct 30 20:19:13.361281 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:13.361202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerDied","Data":"d0342510f3dcdad0c8a46ef504b0e8f8084e154187d8a6d021ced38ad08db77c"}
Oct 30 20:19:14.301023 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:14.300478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:14.301023 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:14.300638    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:14.301023 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:14.300703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:14.301023 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:14.300791    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:14.301023 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:14.300844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:14.301023 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:14.300974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:14.368395 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:14.368354    2577 generic.go:356] "Generic (PLEG): container finished" podID="5151f952d069614d161163f3dcf066f0" containerID="5c06d9929778f39b0ad428c042cbe110a5ffec959b720b888410f4cc155a5f21" exitCode=0
Oct 30 20:19:14.368852 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:14.368455    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"5151f952d069614d161163f3dcf066f0","Type":"ContainerDied","Data":"5c06d9929778f39b0ad428c042cbe110a5ffec959b720b888410f4cc155a5f21"}
Oct 30 20:19:15.394780 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.393967    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"066c09478c1923ae07faa25b9022b5b3","Type":"ContainerStarted","Data":"0c016980246d4779bae81578dc94ceac9fab7ea99c6c633803b39cc04af1395d"}
Oct 30 20:19:15.398104 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.397561    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pxcqk" event={"ID":"4bded661-99cb-4a66-a36b-4e0a446321ef","Type":"ContainerStarted","Data":"482e8500d2fac0679e7c059becff7858607623e6cdd9bb5762217436c10979e8"}
Oct 30 20:19:15.403333 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.402830    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"5151f952d069614d161163f3dcf066f0","Type":"ContainerStarted","Data":"3ec51756baf52f67867d9d6798c389bf98396728dc1b324c6edc42f41dc82666"}
Oct 30 20:19:15.406554 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.406522    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qtnkw" event={"ID":"707f544c-cf0c-4c54-bd0f-6dec8791c59f","Type":"ContainerStarted","Data":"5ee76aeb7a3bbeaffcb920fdee3ec2520ce0225e442387bd415d86026dc0a9f2"}
Oct 30 20:19:15.691326 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.691204    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal" podStartSLOduration=7.691178424 podStartE2EDuration="7.691178424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:15.528363305 +0000 UTC m=+8.725794348" watchObservedRunningTime="2026-10-30 20:19:15.691178424 +0000 UTC m=+8.888609467"
Oct 30 20:19:15.823474 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.822338    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pxcqk" podStartSLOduration=2.542729305 podStartE2EDuration="7.822316167s" totalImagesPullingTime="5.279586862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:15.691743849 +0000 UTC m=+8.889174892" watchObservedRunningTime="2026-10-30 20:19:15.822316167 +0000 UTC m=+9.019747212"
Oct 30 20:19:15.823474 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.822494    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qtnkw" podStartSLOduration=1.601580126 podStartE2EDuration="7.822483352s" totalImagesPullingTime="6.220903226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:15.821936407 +0000 UTC m=+9.019367447" watchObservedRunningTime="2026-10-30 20:19:15.822483352 +0000 UTC m=+9.019914397"
Oct 30 20:19:15.949223 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:15.949034    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" podStartSLOduration=7.949011398 podStartE2EDuration="7.949011398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:15.948666919 +0000 UTC m=+9.146097962" watchObservedRunningTime="2026-10-30 20:19:15.949011398 +0000 UTC m=+9.146442441"
Oct 30 20:19:16.153420 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.153382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:16.153597 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.153437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:16.153665 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.153599    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:16.153739 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.153669    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.153649602 +0000 UTC m=+17.351080622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:16.153845 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.153750    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:16.153845 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.153788    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.153775477 +0000 UTC m=+17.351206498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:16.254381 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.254345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:16.254612 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.254589    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:16.254705 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.254622    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:16.254705 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.254636    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:16.254814 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.254717    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:24.254674608 +0000 UTC m=+17.452105645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:16.300890 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.300858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:16.301041 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.300983    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:16.301126 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.301057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:16.301186 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.301134    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:16.301247 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.301193    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:16.301297 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:16.301283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:16.415099 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.413757    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p" event={"ID":"e97088a0-b646-48d9-818b-c932841333e4","Type":"ContainerStarted","Data":"57cb0971d10e59daba0913cdb6697ac81e13e0fcc58db4f6edc106ce6ff089c5"}
Oct 30 20:19:16.451168 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.451137    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:16.451654 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:16.451629    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:18.248414 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:18.248384    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:19:18.300918 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:18.300892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:18.301051 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:18.300934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:18.301051 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:18.301030    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:18.301174 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:18.300900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:18.301174 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:18.301116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:18.301274 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:18.301203    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:18.420977 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:18.420410    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:18.420977 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:18.420415    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p" event={"ID":"e97088a0-b646-48d9-818b-c932841333e4","Type":"ContainerStarted","Data":"c697ced26f8b5a7057ca383b69da0dd4c91a5daaf85e50b0bbdd2439ce0855b5"}
Oct 30 20:19:19.227531 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.227406    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:19:18.248398214Z","UUID":"f41c9670-f481-4249-a682-68a84bc0352b","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:19:19.229699 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.229644    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:19:19.229824 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.229709    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:19:19.426235 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.426203    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="1476627a0f899533b99dadd2d105a91cb330bea6b2309e955d4a46973106e4c3" exitCode=0
Oct 30 20:19:19.426637 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.426291    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerDied","Data":"1476627a0f899533b99dadd2d105a91cb330bea6b2309e955d4a46973106e4c3"}
Oct 30 20:19:19.428581 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.428512    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dgvmc" event={"ID":"213915b5-832d-44c7-a046-12833d574630","Type":"ContainerStarted","Data":"5b9ac946fbbcacdff34a08d4b1630d775f1ca3a375fbb5abc2a385f4c3f445d7"}
Oct 30 20:19:19.431037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.431014    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cfstj" event={"ID":"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97","Type":"ContainerStarted","Data":"ebad7f46122350410ba7a7c5f2f3c32db7bf7a05a7b56e1160aa7710213dfbd1"}
Oct 30 20:19:19.432920 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.432850    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-blv5n" event={"ID":"b4bac513-58af-4a06-a6e6-f419b8a9bc60","Type":"ContainerStarted","Data":"a1e9f8cf258b58a3c0098db6efb8c42ca5d2835c06bffa486162aba067113d7a"}
Oct 30 20:19:19.754764 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.754547    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:19.754902 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.754884    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:19.755646 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.755623    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:19:19.790312 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.790257    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-blv5n" podStartSLOduration=2.709102703 podStartE2EDuration="11.790244499s" totalImagesPullingTime="9.081141796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:19.668242997 +0000 UTC m=+12.865674035" watchObservedRunningTime="2026-10-30 20:19:19.790244499 +0000 UTC m=+12.987675537"
Oct 30 20:19:19.790474 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:19.790380    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cfstj" podStartSLOduration=2.998047223 podStartE2EDuration="11.790374395s" totalImagesPullingTime="8.792327172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:19.789713725 +0000 UTC m=+12.987144767" watchObservedRunningTime="2026-10-30 20:19:19.790374395 +0000 UTC m=+12.987805433"
Oct 30 20:19:20.029161 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:20.029107    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dgvmc" podStartSLOduration=2.9331962320000002 podStartE2EDuration="12.029093445s" totalImagesPullingTime="9.095897213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:19.907262188 +0000 UTC m=+13.104693237" watchObservedRunningTime="2026-10-30 20:19:20.029093445 +0000 UTC m=+13.226524482"
Oct 30 20:19:20.301036 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:20.300962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:20.301036 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:20.301005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:20.301262 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:20.301074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:20.301430 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:20.301408    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:20.301527 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:20.301468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:20.301592 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:20.301535    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:20.438023 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:20.437988    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p" event={"ID":"e97088a0-b646-48d9-818b-c932841333e4","Type":"ContainerStarted","Data":"a1646c5b44e2383c30aa3090acb82564e4679e225d24d8341367dbf7ad8a8c3f"}
Oct 30 20:19:20.557628 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:20.557523    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p" podStartSLOduration=2.162313644 podStartE2EDuration="12.557508797s" totalImagesPullingTime="10.395195153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:20.557170486 +0000 UTC m=+13.754601525" watchObservedRunningTime="2026-10-30 20:19:20.557508797 +0000 UTC m=+13.754939834"
Oct 30 20:19:21.442093 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:21.442054    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jw4mv" event={"ID":"1f681324-576a-4ae5-95fd-2a7e8f8ce67a","Type":"ContainerStarted","Data":"c88762155c4bc1e869e0b2d5f07e34dba80e94c03d48f2ccfb3400374a938841"}
Oct 30 20:19:21.444664 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:21.444634    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="fe904096cdc3de4e910e5139b02af381656646c68855cce2389d462ee991ebe8" exitCode=0
Oct 30 20:19:21.444796 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:21.444749    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerDied","Data":"fe904096cdc3de4e910e5139b02af381656646c68855cce2389d462ee991ebe8"}
Oct 30 20:19:21.565035 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:21.564985    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jw4mv" podStartSLOduration=2.6620887939999998 podStartE2EDuration="13.564970006s" totalImagesPullingTime="10.902881212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:21.564525098 +0000 UTC m=+14.761956138" watchObservedRunningTime="2026-10-30 20:19:21.564970006 +0000 UTC m=+14.762401073"
Oct 30 20:19:22.301037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:22.301007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:22.301200 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:22.301046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:22.301200 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:22.301118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:22.301284 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:22.301237    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:22.301356 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:22.301323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:22.301456 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:22.301424    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:23.451759 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:23.451713    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="26538b2c1fa4e37c90567af263c5a4bdf1c9ec35e0c7f95ebc8f9ea927f49c31" exitCode=0
Oct 30 20:19:23.452443 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:23.451791    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerDied","Data":"26538b2c1fa4e37c90567af263c5a4bdf1c9ec35e0c7f95ebc8f9ea927f49c31"}
Oct 30 20:19:24.219585 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:24.219551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:24.219585 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:24.219590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:24.219816 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.219710    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:24.219816 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.219714    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:24.219816 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.219785    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:40.219772129 +0000 UTC m=+33.417203146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:24.219816 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.219800    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:19:40.219793354 +0000 UTC m=+33.417224370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:24.300488 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:24.300450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:24.300488 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:24.300472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:24.300719 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:24.300450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:24.300719 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.300558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:24.300719 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.300658    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:24.300871 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.300776    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:24.320836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:24.320801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:24.320998 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.320980    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:24.321045 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.321004    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:24.321045 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.321018    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:24.321144 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:24.321079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:40.321060125 +0000 UTC m=+33.518491159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:25.460755 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:25.460478    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"e9a27244376878ae12032185ba1823a4de2b356cfc15c633840a508dbe4333f3"}
Oct 30 20:19:25.461195 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:25.460812    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"15fd5869286ec3ce318518596e8aba20d1a39b099cd49bf413568243ed8de930"}
Oct 30 20:19:25.461195 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:25.460865    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"92fdbe647bfde7c16ce04ae9ee44101d7038da5466b1562b08bc422ca6b3ce0c"}
Oct 30 20:19:25.461195 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:25.460912    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"36849327f63037a250aa18f50d86b2b174b794785836130e8a7b6b5ea7aa12ff"}
Oct 30 20:19:25.461195 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:25.460957    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"93ffd8dc58dc735820697b79c3949d164bfa2e452ebff4f18c3081d4c80c97e2"}
Oct 30 20:19:25.461195 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:25.461001    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"8e25c9cab0f6c432abc6a4eeff9bf812e182a3c95af35a8cc9bbb28489d15107"}
Oct 30 20:19:26.300128 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:26.299944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:26.300287 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:26.299986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:26.300287 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:26.300198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:26.300287 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:26.299995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:26.300287 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:26.300274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:26.300439 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:26.300361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:26.464957 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:26.464931    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="48770a6fe7de3f9864b4505c69799be75fb14705c75fa97c9cde8571ee4fc197" exitCode=0
Oct 30 20:19:26.465311 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:26.464980    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerDied","Data":"48770a6fe7de3f9864b4505c69799be75fb14705c75fa97c9cde8571ee4fc197"}
Oct 30 20:19:27.470971 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:27.470941    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="bf781a741af3599d8334f700f15cee9dc0936e239588f21d984990646b6c5ee2" exitCode=0
Oct 30 20:19:27.471437 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:27.471008    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerDied","Data":"bf781a741af3599d8334f700f15cee9dc0936e239588f21d984990646b6c5ee2"}
Oct 30 20:19:27.474628 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:27.474611    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"2f3b5f977d5be64975b44395f54b493963489d6797d3f3a6a0601216137196c0"}
Oct 30 20:19:28.300733 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:28.300699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:28.300733 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:28.300721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:28.300950 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:28.300813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:28.300950 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:28.300828    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:28.300950 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:28.300877    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:28.301037 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:28.300948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:28.479798 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:28.479758    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rk542" event={"ID":"2f19916b-62f3-4a81-ab47-4cde132463c5","Type":"ContainerStarted","Data":"99eb2b4569d5a900f37881607688968d08d5b6eee06e3e41a4b5d67dc349fe34"}
Oct 30 20:19:28.605912 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:28.605833    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rk542" podStartSLOduration=6.979953641 podStartE2EDuration="20.605820134s" totalImagesPullingTime="13.625866493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:28.605529484 +0000 UTC m=+21.802960522" watchObservedRunningTime="2026-10-30 20:19:28.605820134 +0000 UTC m=+21.803251172"
Oct 30 20:19:29.486260 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:29.486223    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" event={"ID":"c9789942-84f9-4af9-ba2d-52d3115b5882","Type":"ContainerStarted","Data":"de7a6b471073903faa502435d568e18bb9ada2d04322ddb03d28549b0a21dc02"}
Oct 30 20:19:29.769468 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:29.769422    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42" podStartSLOduration=5.673294428 podStartE2EDuration="21.769409236s" totalImagesPullingTime="16.096114808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:29.718638601 +0000 UTC m=+22.916069639" watchObservedRunningTime="2026-10-30 20:19:29.769409236 +0000 UTC m=+22.966840274"
Oct 30 20:19:30.301003 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.300976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:30.301218 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.301020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:30.301218 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.301031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:30.301218 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:30.301102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:30.301218 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:30.301158    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:30.301342 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:30.301224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:30.488566 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.488525    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:30.489103 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.488614    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:30.489103 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.488644    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:30.504171 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.504142    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:30.504312 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:30.504240    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:19:32.152958 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:32.152614    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tj2kt"]
Oct 30 20:19:32.153382 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:32.152976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:32.153382 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:32.153086    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:32.153382 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:32.153275    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8ss9g"]
Oct 30 20:19:32.153490 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:32.153388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:32.153490 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:32.153483    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:32.153903 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:32.153882    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f84bz"]
Oct 30 20:19:32.153984 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:32.153958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:32.154049 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:32.154032    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:33.300648 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:33.300616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:33.301173 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:33.300755    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:34.094467 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:34.094436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pxcqk_4bded661-99cb-4a66-a36b-4e0a446321ef/konnectivity-agent/0.log" limit=1024
Oct 30 20:19:34.300938 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:34.300894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:34.301305 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:34.300896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:34.301305 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:34.301023    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:34.301305 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:34.301085    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:35.300668 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:35.300634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:35.300838 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:35.300793    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:36.300102 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:36.300065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:36.300445 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:36.300071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:36.300445 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:36.300168    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:36.300445 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:36.300252    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:37.301024 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:37.300996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:37.301395 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:37.301085    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:38.300260 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:38.300232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:38.300260 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:38.300261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:38.300449 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:38.300335    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:38.300484 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:38.300451    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:39.300248 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:39.300219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:39.300618 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:39.300312    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:40.240929 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:40.240891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:40.240929 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:40.240932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:40.241118 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.241032    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:40.241118 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.241045    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:40.241118 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.241089    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs podName:cb0e115f-349e-4f69-b643-336b58e3c56c nodeName:}" failed. No retries permitted until 2026-10-30 20:20:12.241074837 +0000 UTC m=+65.438505854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs") pod "network-metrics-daemon-8ss9g" (UID: "cb0e115f-349e-4f69-b643-336b58e3c56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:40.241118 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.241102    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret podName:f489b9e7-d156-4369-9afc-b38d675d1a31 nodeName:}" failed. No retries permitted until 2026-10-30 20:20:12.241096596 +0000 UTC m=+65.438527612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret") pod "global-pull-secret-syncer-f84bz" (UID: "f489b9e7-d156-4369-9afc-b38d675d1a31") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:40.300896 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:40.300869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:40.301248 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.300993    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:40.301248 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:40.301020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:40.301248 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.301080    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:40.342090 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:40.342060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:40.342200 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.342170    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:40.342200 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.342183    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:40.342200 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.342191    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bktlv for pod openshift-network-diagnostics/network-check-target-tj2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:40.342314 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:40.342244    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv podName:de0184b4-b27e-4bcf-8a5f-dacb354d2574 nodeName:}" failed. No retries permitted until 2026-10-30 20:20:12.342227729 +0000 UTC m=+65.539658759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bktlv" (UniqueName: "kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv") pod "network-check-target-tj2kt" (UID: "de0184b4-b27e-4bcf-8a5f-dacb354d2574") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:40.355808 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:40.355788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-blv5n_b4bac513-58af-4a06-a6e6-f419b8a9bc60/dns-node-resolver/0.log"
Oct 30 20:19:41.300198 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:41.300171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:41.300360 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:41.300284    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:41.358550 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:41.358526    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qtnkw_707f544c-cf0c-4c54-bd0f-6dec8791c59f/node-ca/0.log"
Oct 30 20:19:42.300704 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:42.300670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:42.300864 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:42.300678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:42.300864 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:42.300788    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:42.300864 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:42.300851    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:43.300209 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:43.300182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:43.300570 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:43.300270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:44.300979 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:44.300949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:44.301358 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:44.300949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:44.301358 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:44.301053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:44.301358 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:44.301118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:45.300824 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:45.300785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:45.301002 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:45.300919    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:46.300672 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:46.300641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:46.300948 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:46.300796    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:46.300948 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:46.300849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:46.300948 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:46.300932    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:47.301399 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:47.301366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:47.301811 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:47.301450    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:48.300051 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:48.300010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:48.300051 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:48.300025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:48.300297 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:48.300124    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:48.300297 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:48.300214    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:49.300324 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:49.300289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:49.300761 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:49.300412    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:50.300016 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:50.299974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:50.300206 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:50.299980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:50.300206 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:50.300089    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:50.300330 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:50.300217    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:51.300938 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:51.300901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:51.301323 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:51.301013    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:52.300404 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:52.300376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:52.300561 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:52.300376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:52.300561 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:52.300471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f84bz" podUID="f489b9e7-d156-4369-9afc-b38d675d1a31"
Oct 30 20:19:52.300634 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:52.300564    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ss9g" podUID="cb0e115f-349e-4f69-b643-336b58e3c56c"
Oct 30 20:19:53.300343 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.300311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:53.300679 ip-10-0-130-195 kubenswrapper[2577]: E1030 20:19:53.300436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tj2kt" podUID="de0184b4-b27e-4bcf-8a5f-dacb354d2574"
Oct 30 20:19:53.476580 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.476555    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:19:53.476723 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.476667    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:19:53.633370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.633333    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b42wp"]
Oct 30 20:19:53.646257 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.646234    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9zjw5"]
Oct 30 20:19:53.646370 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.646354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.660260 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.660238    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9nxm"]
Oct 30 20:19:53.660392 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.660375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.672353 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.672336    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9zjw5"]
Oct 30 20:19:53.672435 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.672371    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b42wp"]
Oct 30 20:19:53.672475 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.672444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:53.679048 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.679027    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9nxm"]
Oct 30 20:19:53.702909 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.702885    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:19:53.703135 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.703120    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q6sjr\""
Oct 30 20:19:53.703200 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.703121    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:19:53.720016 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.719995    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:19:53.720146 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.720134    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:19:53.720422 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.720409    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:19:53.720422 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.720413    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4ct2\""
Oct 30 20:19:53.720520 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.720421    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:19:53.736819 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.736774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95c43767-28f8-4bfc-960c-fcf1ee6c0207-config-volume\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.736819 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.736801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdq4\" (UniqueName: \"kubernetes.io/projected/95c43767-28f8-4bfc-960c-fcf1ee6c0207-kube-api-access-6rdq4\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.736917 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.736834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c43767-28f8-4bfc-960c-fcf1ee6c0207-metrics-tls\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.736971 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.736957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c43767-28f8-4bfc-960c-fcf1ee6c0207-tmp-dir\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.749705 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.749684    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:19:53.749806 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.749716    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bp5wk\""
Oct 30 20:19:53.749806 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.749721    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:19:53.749806 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.749765    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:19:53.837692 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c43767-28f8-4bfc-960c-fcf1ee6c0207-metrics-tls\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.837822 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75417679-7317-484c-bc01-0d7870ae7825-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.837822 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75417679-7317-484c-bc01-0d7870ae7825-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.837822 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8z2\" (UniqueName: \"kubernetes.io/projected/75417679-7317-484c-bc01-0d7870ae7825-kube-api-access-6b8z2\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.837822 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9185192-48b5-4f1d-9e4c-8344f4c64058-cert\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:53.837967 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrfg\" (UniqueName: \"kubernetes.io/projected/d9185192-48b5-4f1d-9e4c-8344f4c64058-kube-api-access-fjrfg\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:53.837967 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c43767-28f8-4bfc-960c-fcf1ee6c0207-tmp-dir\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.837967 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75417679-7317-484c-bc01-0d7870ae7825-crio-socket\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.838063 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.837982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75417679-7317-484c-bc01-0d7870ae7825-data-volume\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.838063 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.838034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95c43767-28f8-4bfc-960c-fcf1ee6c0207-config-volume\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.838135 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.838097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c43767-28f8-4bfc-960c-fcf1ee6c0207-tmp-dir\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.838135 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.838124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdq4\" (UniqueName: \"kubernetes.io/projected/95c43767-28f8-4bfc-960c-fcf1ee6c0207-kube-api-access-6rdq4\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.838476 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.838458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95c43767-28f8-4bfc-960c-fcf1ee6c0207-config-volume\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.841331 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.841312    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c43767-28f8-4bfc-960c-fcf1ee6c0207-metrics-tls\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.896821 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.896800    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdq4\" (UniqueName: \"kubernetes.io/projected/95c43767-28f8-4bfc-960c-fcf1ee6c0207-kube-api-access-6rdq4\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.938853 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.938830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9185192-48b5-4f1d-9e4c-8344f4c64058-cert\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:53.938945 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.938859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrfg\" (UniqueName: \"kubernetes.io/projected/d9185192-48b5-4f1d-9e4c-8344f4c64058-kube-api-access-fjrfg\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:53.938945 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.938902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75417679-7317-484c-bc01-0d7870ae7825-crio-socket\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939024 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75417679-7317-484c-bc01-0d7870ae7825-data-volume\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939059 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939049    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75417679-7317-484c-bc01-0d7870ae7825-crio-socket\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939118 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75417679-7317-484c-bc01-0d7870ae7825-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939155 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75417679-7317-484c-bc01-0d7870ae7825-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939192 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8z2\" (UniqueName: \"kubernetes.io/projected/75417679-7317-484c-bc01-0d7870ae7825-kube-api-access-6b8z2\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939372 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939344    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75417679-7317-484c-bc01-0d7870ae7825-data-volume\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.939609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.939594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75417679-7317-484c-bc01-0d7870ae7825-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.941180 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.941158    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9185192-48b5-4f1d-9e4c-8344f4c64058-cert\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:53.941330 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.941315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75417679-7317-484c-bc01-0d7870ae7825-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:53.981449 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.981428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:53.998480 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:53.998436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrfg\" (UniqueName: \"kubernetes.io/projected/d9185192-48b5-4f1d-9e4c-8344f4c64058-kube-api-access-fjrfg\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:54.000569 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.000553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8z2\" (UniqueName: \"kubernetes.io/projected/75417679-7317-484c-bc01-0d7870ae7825-kube-api-access-6b8z2\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:54.033233 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.033199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:19:54.051448 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.051425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:19:54.301068 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.300992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:19:54.301068 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.301039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:19:54.357458 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.357441    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:19:54.358843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.358824    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:19:54.358843 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.358829    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:19:54.358986 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.358879    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:19:54.424891 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.424864    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b42wp"]
Oct 30 20:19:54.483408 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.483377    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9nxm"]
Oct 30 20:19:54.486417 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:54.486380    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9185192_48b5_4f1d_9e4c_8344f4c64058.slice/crio-eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90 WatchSource:0}: Error finding container eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90: Status 404 returned error can't find the container with id eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90
Oct 30 20:19:54.491060 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.491039    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9zjw5"]
Oct 30 20:19:54.498648 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:54.498628    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75417679_7317_484c_bc01_0d7870ae7825.slice/crio-0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf WatchSource:0}: Error finding container 0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf: Status 404 returned error can't find the container with id 0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf
Oct 30 20:19:54.540054 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.540016    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zjw5" event={"ID":"75417679-7317-484c-bc01-0d7870ae7825","Type":"ContainerStarted","Data":"0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf"}
Oct 30 20:19:54.541210 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.541181    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9nxm" event={"ID":"d9185192-48b5-4f1d-9e4c-8344f4c64058","Type":"ContainerStarted","Data":"eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90"}
Oct 30 20:19:54.542172 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:54.542154    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b42wp" event={"ID":"95c43767-28f8-4bfc-960c-fcf1ee6c0207","Type":"ContainerStarted","Data":"caaebb157709cbd7c674118147873bdc6345f4a844bd05dee7dd87cd7256c469"}
Oct 30 20:19:55.300543 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:55.300507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:19:55.355098 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:55.355071    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:19:55.355098 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:55.355071    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:19:55.355477 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:55.355125    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:19:55.547077 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:55.547039    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zjw5" event={"ID":"75417679-7317-484c-bc01-0d7870ae7825","Type":"ContainerStarted","Data":"08a9947a13026034326bf23fa62a84749ca12f7a5ea5532a6970c5fe3aa3faf3"}
Oct 30 20:19:56.553146 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:56.553110    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zjw5" event={"ID":"75417679-7317-484c-bc01-0d7870ae7825","Type":"ContainerStarted","Data":"3c050ec31eb7a97528b283422368707351e816c5ac838e11f052cf7b42d3fe25"}
Oct 30 20:19:56.953316 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:56.953215    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"]
Oct 30 20:19:56.956554 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:56.955734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:57.013079 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.013048    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"]
Oct 30 20:19:57.062480 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.062443    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z59pm\""
Oct 30 20:19:57.062649 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.062502    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 20:19:57.062729 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.062700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/de295ab7-6640-4744-826b-e780a3d7e597-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nnfbg\" (UID: \"de295ab7-6640-4744-826b-e780a3d7e597\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:57.114956 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.114010    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f877b658-k6n9x"]
Oct 30 20:19:57.116061 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.116032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.163318 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.163278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/de295ab7-6640-4744-826b-e780a3d7e597-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nnfbg\" (UID: \"de295ab7-6640-4744-826b-e780a3d7e597\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:57.166459 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.166432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/de295ab7-6640-4744-826b-e780a3d7e597-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nnfbg\" (UID: \"de295ab7-6640-4744-826b-e780a3d7e597\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:57.170046 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170014    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 20:19:57.170181 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170134    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 20:19:57.170255 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170236    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 20:19:57.170397 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170380    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fx2fq\""
Oct 30 20:19:57.170465 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170390    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 20:19:57.170660 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170608    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 20:19:57.170660 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170483    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 20:19:57.170810 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.170452    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 20:19:57.176080 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.176031    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f877b658-k6n9x"]
Oct 30 20:19:57.263831 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.263791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.263831 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.263841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.264054 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.263865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.264054 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.263911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.264054 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.264001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.264054 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.264041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.264054 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.264057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.272358 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.272327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:57.364546 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.364679 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.364679 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.364679 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.364847 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.364847 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.364847 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.364773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.365915 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.365861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.366420 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.366398    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.366677 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.366657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.367225 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.367202    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.369576 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.369530    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.370133 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.370112    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.425618 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.425586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.428269 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.428242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:19:57.559855 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.558970    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b42wp" event={"ID":"95c43767-28f8-4bfc-960c-fcf1ee6c0207","Type":"ContainerStarted","Data":"962b3da2d53c5622213776d630990aafb60ce6c22b94bc216f769833f035f11b"}
Oct 30 20:19:57.559855 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.559040    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b42wp" event={"ID":"95c43767-28f8-4bfc-960c-fcf1ee6c0207","Type":"ContainerStarted","Data":"55a83a71a2efb471f9edbe04a38bdf20eafdd017a578a83e176a2e9d78bb1030"}
Oct 30 20:19:57.559855 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.559804    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b42wp"
Oct 30 20:19:57.653160 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.653086    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"]
Oct 30 20:19:57.657331 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:57.657299    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde295ab7_6640_4744_826b_e780a3d7e597.slice/crio-ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599 WatchSource:0}: Error finding container ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599: Status 404 returned error can't find the container with id ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599
Oct 30 20:19:57.676018 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.675977    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-b42wp" podStartSLOduration=2.470022302 podStartE2EDuration="4.675963239s" totalImagesPullingTime="2.205940937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:57.675677769 +0000 UTC m=+50.873108806" watchObservedRunningTime="2026-10-30 20:19:57.675963239 +0000 UTC m=+50.873394275"
Oct 30 20:19:57.821568 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:57.821534    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f877b658-k6n9x"]
Oct 30 20:19:57.824844 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:19:57.824820    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ef5c28_b16d_425a_b173_0ec514a3c3de.slice/crio-71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2 WatchSource:0}: Error finding container 71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2: Status 404 returned error can't find the container with id 71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2
Oct 30 20:19:58.562596 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:58.562560    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f877b658-k6n9x" event={"ID":"51ef5c28-b16d-425a-b173-0ec514a3c3de","Type":"ContainerStarted","Data":"71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2"}
Oct 30 20:19:58.563861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:58.563838    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg" event={"ID":"de295ab7-6640-4744-826b-e780a3d7e597","Type":"ContainerStarted","Data":"ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599"}
Oct 30 20:19:58.565968 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:58.565939    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zjw5" event={"ID":"75417679-7317-484c-bc01-0d7870ae7825","Type":"ContainerStarted","Data":"fc4df965d29b4c5a927af149f98998b0e1bdae05e6ffbedf868e0363b4d92a7a"}
Oct 30 20:19:58.567398 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:58.567377    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9nxm" event={"ID":"d9185192-48b5-4f1d-9e4c-8344f4c64058","Type":"ContainerStarted","Data":"71d35d50a66a053b2c8f1775cd5be09c8a7fd35f9153d13e7f2b50f1d00ad186"}
Oct 30 20:19:58.683795 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:58.683728    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9zjw5" podStartSLOduration=2.664005483 podStartE2EDuration="5.683712248s" totalImagesPullingTime="3.019706765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:58.68321293 +0000 UTC m=+51.880643969" watchObservedRunningTime="2026-10-30 20:19:58.683712248 +0000 UTC m=+51.881143285"
Oct 30 20:19:58.802194 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:58.802141    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9nxm" podStartSLOduration=2.085500473 podStartE2EDuration="5.802130243s" totalImagesPullingTime="3.71662977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:58.801626071 +0000 UTC m=+51.999057115" watchObservedRunningTime="2026-10-30 20:19:58.802130243 +0000 UTC m=+51.999561279"
Oct 30 20:19:59.574796 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:59.574755    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg" event={"ID":"de295ab7-6640-4744-826b-e780a3d7e597","Type":"ContainerStarted","Data":"cf2b94c3b72dd825aca962777f254c7c20a29264455960ad35d623c3fe540087"}
Oct 30 20:19:59.576564 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:59.576028    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:59.581303 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:59.581275    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:19:59.692522 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:19:59.692457    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg" podStartSLOduration=2.168296299 podStartE2EDuration="3.692443429s" totalImagesPullingTime="1.52414713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:59.69153455 +0000 UTC m=+52.888965587" watchObservedRunningTime="2026-10-30 20:19:59.692443429 +0000 UTC m=+52.889874465"
Oct 30 20:20:00.542341 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.542312    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-f424d"]
Oct 30 20:20:00.545487 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.545466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.602769 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.602718    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 20:20:00.653153 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.653118    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 20:20:00.653153 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.653129    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:20:00.653363 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.653181    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tcbkk\""
Oct 30 20:20:00.653846 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.653809    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:20:00.653942 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.653857    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:20:00.689725 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.689695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.689725 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.689726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d81e86-e9cb-4bf1-9286-74af62e818ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.689929 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.689792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cv52\" (UniqueName: \"kubernetes.io/projected/79d81e86-e9cb-4bf1-9286-74af62e818ee-kube-api-access-8cv52\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.689929 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.689851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.704904 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.704878    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-f424d"]
Oct 30 20:20:00.790939 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.790895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.791117 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.790944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d81e86-e9cb-4bf1-9286-74af62e818ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.791176 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.791133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cv52\" (UniqueName: \"kubernetes.io/projected/79d81e86-e9cb-4bf1-9286-74af62e818ee-kube-api-access-8cv52\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.791229 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.791211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.791725 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.791668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d81e86-e9cb-4bf1-9286-74af62e818ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.793901 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.793844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.794367 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.794347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.848954 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.848925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cv52\" (UniqueName: \"kubernetes.io/projected/79d81e86-e9cb-4bf1-9286-74af62e818ee-kube-api-access-8cv52\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:00.859650 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:00.859622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:01.239030 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:01.238999    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-f424d"]
Oct 30 20:20:01.272421 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:01.272394    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d81e86_e9cb_4bf1_9286_74af62e818ee.slice/crio-7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326 WatchSource:0}: Error finding container 7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326: Status 404 returned error can't find the container with id 7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326
Oct 30 20:20:01.582170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:01.582085    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d" event={"ID":"79d81e86-e9cb-4bf1-9286-74af62e818ee","Type":"ContainerStarted","Data":"7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326"}
Oct 30 20:20:01.583625 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:01.583600    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f877b658-k6n9x" event={"ID":"51ef5c28-b16d-425a-b173-0ec514a3c3de","Type":"ContainerStarted","Data":"4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73"}
Oct 30 20:20:01.705562 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:01.705513    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76f877b658-k6n9x" podStartSLOduration=1.5750108790000001 podStartE2EDuration="4.705499458s" totalImagesPullingTime="3.130488579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:01.70482333 +0000 UTC m=+54.902254368" watchObservedRunningTime="2026-10-30 20:20:01.705499458 +0000 UTC m=+54.902930496"
Oct 30 20:20:02.505852 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:02.505808    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:03.592569 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:03.592526    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d" event={"ID":"79d81e86-e9cb-4bf1-9286-74af62e818ee","Type":"ContainerStarted","Data":"76fa51ebfceaffcf81665c2e39cc39b62053154d6a7acbbddf7f215f79919370"}
Oct 30 20:20:03.593046 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:03.592589    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d" event={"ID":"79d81e86-e9cb-4bf1-9286-74af62e818ee","Type":"ContainerStarted","Data":"1a9a0e2c6f671e3a5d607d1a2b7357e7016b25c91829e14fa8ac23ff70da8aef"}
Oct 30 20:20:03.718965 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:03.718914    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d" podStartSLOduration=1.894847715 podStartE2EDuration="3.718899618s" totalImagesPullingTime="1.824051903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:03.718120681 +0000 UTC m=+56.915551718" watchObservedRunningTime="2026-10-30 20:20:03.718899618 +0000 UTC m=+56.916330654"
Oct 30 20:20:06.957394 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:06.957363    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"]
Oct 30 20:20:06.959522 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:06.959507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.015552 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.015523    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:20:07.035972 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.035939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.036142 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.035983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9sm\" (UniqueName: \"kubernetes.io/projected/ec59824c-8bdf-4099-9124-26a29ae8e876-kube-api-access-6j9sm\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.036142 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.036012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec59824c-8bdf-4099-9124-26a29ae8e876-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.036251 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.036174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.065635 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.065516    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 20:20:07.065635 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.065622    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h255w\""
Oct 30 20:20:07.068135 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.068108    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"]
Oct 30 20:20:07.075134 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.075106    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"]
Oct 30 20:20:07.077563 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.077547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.117637 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.117603    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l9bdp"]
Oct 30 20:20:07.119541 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.119520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.130819 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.130802    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:20:07.131060 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.131045    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 20:20:07.131060 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.131056    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 20:20:07.131136 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.131076    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-svhlw\""
Oct 30 20:20:07.137465 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-textfile\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.137568 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.137568 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-wtmp\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.137568 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.137721 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-tls\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-root\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.137917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w25b\" (UniqueName: \"kubernetes.io/projected/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-api-access-4w25b\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-metrics-client-ca\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec59824c-8bdf-4099-9124-26a29ae8e876-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqzd\" (UniqueName: \"kubernetes.io/projected/81ab7c91-f29a-4204-b268-5407d7129914-kube-api-access-jtqzd\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9sm\" (UniqueName: \"kubernetes.io/projected/ec59824c-8bdf-4099-9124-26a29ae8e876-kube-api-access-6j9sm\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.138881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-sys\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.139388 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.139045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec59824c-8bdf-4099-9124-26a29ae8e876-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.140931 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.140907    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"]
Oct 30 20:20:07.140931 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.140920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.141092 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.141026    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.196661 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.196633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9sm\" (UniqueName: \"kubernetes.io/projected/ec59824c-8bdf-4099-9124-26a29ae8e876-kube-api-access-6j9sm\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.231089 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.231063    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9jp6\""
Oct 30 20:20:07.231292 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.231271    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:20:07.231368 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.231314    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:20:07.231413 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.231377    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:20:07.239245 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-root\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239360 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.239360 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w25b\" (UniqueName: \"kubernetes.io/projected/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-api-access-4w25b\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.239360 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-metrics-client-ca\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239360 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-root\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239360 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.239608 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239608 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239608 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.239608 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.239608 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqzd\" (UniqueName: \"kubernetes.io/projected/81ab7c91-f29a-4204-b268-5407d7129914-kube-api-access-jtqzd\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239877 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-sys\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239877 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-textfile\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.239877 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-wtmp\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240022 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-sys\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240022 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.240022 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.239967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-tls\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.240025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-textfile\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.240041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-wtmp\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.240079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-metrics-client-ca\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.240081    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.240170 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.240081    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.240385 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.240358    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.242254 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.242233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.242489 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.242466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-tls\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.291031 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.291008    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 20:20:07.291211 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.291184    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 20:20:07.291321 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.291306    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:20:07.300121 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.300095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqzd\" (UniqueName: \"kubernetes.io/projected/81ab7c91-f29a-4204-b268-5407d7129914-kube-api-access-jtqzd\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.300282 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.300264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w25b\" (UniqueName: \"kubernetes.io/projected/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-api-access-4w25b\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.300361 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.300314    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.301976 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.301954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.302071 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.302005    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.323025 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.323006    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h255w\""
Oct 30 20:20:07.331399 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.331379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:07.429058 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.429023    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:07.429208 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.429114    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:07.430947 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.430929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:07.433549 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.433509    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:07.440560 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:07.440537    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ab7c91_f29a_4204_b268_5407d7129914.slice/crio-ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8 WatchSource:0}: Error finding container ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8: Status 404 returned error can't find the container with id ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8
Oct 30 20:20:07.441963 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.441913    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-svhlw\""
Oct 30 20:20:07.450994 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.450978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:07.605456 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.605420    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9bdp" event={"ID":"81ab7c91-f29a-4204-b268-5407d7129914","Type":"ContainerStarted","Data":"ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8"}
Oct 30 20:20:07.609362 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.609339    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:07.749216 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.749191    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"]
Oct 30 20:20:07.751477 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:07.751448    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec59824c_8bdf_4099_9124_26a29ae8e876.slice/crio-b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82 WatchSource:0}: Error finding container b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82: Status 404 returned error can't find the container with id b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82
Oct 30 20:20:07.841634 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:07.841612    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"]
Oct 30 20:20:07.843393 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:07.843371    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda968a685_34c9_4b22_9bd6_0963e6f2dc5a.slice/crio-cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19 WatchSource:0}: Error finding container cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19: Status 404 returned error can't find the container with id cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19
Oct 30 20:20:08.343485 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.343441    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c79948c48-z5442"]
Oct 30 20:20:08.346127 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.346106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.448764 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.448713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.448933 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.448857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.448933 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.448901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.449028 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.448937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.449028 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.448999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.449090 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.449042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.449090 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.449076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.449152 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.449091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.449939 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.449914    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 20:20:08.451125 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.451103    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c79948c48-z5442"]
Oct 30 20:20:08.550228 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550228 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550422 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550475 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550512 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550562 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550610 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.550651 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.550610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.551124 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.551068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.551439 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.551417    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.551532 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.551489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.551631 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.551606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.551832 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.551809    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.553193 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.553171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.553262 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.553190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.579251 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.579225    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:08.608714 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.608647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:08.617159 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.617123    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b" event={"ID":"a968a685-34c9-4b22-9bd6-0963e6f2dc5a","Type":"ContainerStarted","Data":"cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19"}
Oct 30 20:20:08.620519 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.620494    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6" event={"ID":"ec59824c-8bdf-4099-9124-26a29ae8e876","Type":"ContainerStarted","Data":"8220223b8aba4018a21d2667b0dbc370584b83d65a99338fc164ca9babbb9695"}
Oct 30 20:20:08.620627 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.620537    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6" event={"ID":"ec59824c-8bdf-4099-9124-26a29ae8e876","Type":"ContainerStarted","Data":"44e26b16d9ee67685adcc973f52b79794b93d3edcb85b8798f88e44f059b54d5"}
Oct 30 20:20:08.620627 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.620557    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6" event={"ID":"ec59824c-8bdf-4099-9124-26a29ae8e876","Type":"ContainerStarted","Data":"b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82"}
Oct 30 20:20:08.658838 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:08.658795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:09.040426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.040392    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c79948c48-z5442"]
Oct 30 20:20:09.045074 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:09.045047    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020e1e00_032c_4533_ba58_97b7f4971d37.slice/crio-da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f WatchSource:0}: Error finding container da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f: Status 404 returned error can't find the container with id da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f
Oct 30 20:20:09.355924 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.354397    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:20:09.358643 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.358616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.414572 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414545    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:20:09.414836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414643    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:20:09.414836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414670    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:20:09.414836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414707    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:20:09.414836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414727    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:20:09.414836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414806    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:20:09.414836 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414546    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:20:09.415144 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414854    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:20:09.415144 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.414946    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ldxdn\""
Oct 30 20:20:09.415671 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.415644    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:20:09.425184 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.425161    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:20:09.459621 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.459757 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.459757 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.459857 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.459857 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.459925 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.459925 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.460008 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.459957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.460047 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.460023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.460132 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.460113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.460187 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.460149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.460187 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.460175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.460271 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.460200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.560807 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.560807 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.560979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.561034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561081 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.561063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561462 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.561092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561462 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.561129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561462 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.561153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.561462 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.561181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.562429 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.562022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.562429 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.562078    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.562429 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.562393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.565331 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.565307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.565656 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.565633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.566087 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.566065    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.566169 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.566143    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.567769 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.566624    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.567769 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.566923    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.567769 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.567087    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.567769 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.567152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.567769 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.567188    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.625907 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.625877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.627098 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.627078    2577 generic.go:356] "Generic (PLEG): container finished" podID="81ab7c91-f29a-4204-b268-5407d7129914" containerID="34e81a7227ba332d4855df2601cda81a275194f1b0b649fa83493c4c22c2f49e" exitCode=0
Oct 30 20:20:09.627183 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.627157    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9bdp" event={"ID":"81ab7c91-f29a-4204-b268-5407d7129914","Type":"ContainerDied","Data":"34e81a7227ba332d4855df2601cda81a275194f1b0b649fa83493c4c22c2f49e"}
Oct 30 20:20:09.629226 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.629206    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6" event={"ID":"ec59824c-8bdf-4099-9124-26a29ae8e876","Type":"ContainerStarted","Data":"7f5791386b6e8fed92da959d45c8fabeda751655f2d30c0d0390dc11bbfc6162"}
Oct 30 20:20:09.631120 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.631096    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c79948c48-z5442" event={"ID":"020e1e00-032c-4533-ba58-97b7f4971d37","Type":"ContainerStarted","Data":"96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658"}
Oct 30 20:20:09.631213 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.631136    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c79948c48-z5442" event={"ID":"020e1e00-032c-4533-ba58-97b7f4971d37","Type":"ContainerStarted","Data":"da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f"}
Oct 30 20:20:09.684092 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.683634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:09.692344 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.691603    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c64476cfd-sv4hq"]
Oct 30 20:20:09.701948 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.701929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764055 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.763051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f575738a-d51b-460e-9ad5-e2aeb948f4bc-metrics-client-ca\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764055 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.763895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-grpc-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764055 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.763962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764443 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.764417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.764454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.764492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f575738a-d51b-460e-9ad5-e2aeb948f4bc-tmp\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.764529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.764550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxlb\" (UniqueName: \"kubernetes.io/projected/f575738a-d51b-460e-9ad5-e2aeb948f4bc-kube-api-access-qhxlb\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.764609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.764566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.778586 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.778552    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 20:20:09.778855 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.778831    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 20:20:09.779132 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.779115    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 20:20:09.781027 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.781007    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 20:20:09.781130 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.781058    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zjnsc\""
Oct 30 20:20:09.782611 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.782582    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-199bhgmkq2vek\""
Oct 30 20:20:09.784093 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.784074    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 20:20:09.801112 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.801090    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c64476cfd-sv4hq"]
Oct 30 20:20:09.865052 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.864981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-grpc-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865052 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865238 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865238 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865238 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f575738a-d51b-460e-9ad5-e2aeb948f4bc-tmp\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865368 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865414 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxlb\" (UniqueName: \"kubernetes.io/projected/f575738a-d51b-460e-9ad5-e2aeb948f4bc-kube-api-access-qhxlb\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865464 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865518 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f575738a-d51b-460e-9ad5-e2aeb948f4bc-metrics-client-ca\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.865575 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.865560    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f575738a-d51b-460e-9ad5-e2aeb948f4bc-tmp\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.866444 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.866412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f575738a-d51b-460e-9ad5-e2aeb948f4bc-metrics-client-ca\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.867900 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.867875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.868266 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.868235    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.868348 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.868281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-grpc-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.868411 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.868394    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.868411 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.868400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.868497 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.868437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:09.905087 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.905048    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c79948c48-z5442" podStartSLOduration=1.905035561 podStartE2EDuration="1.905035561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:09.904480467 +0000 UTC m=+63.101911505" watchObservedRunningTime="2026-10-30 20:20:09.905035561 +0000 UTC m=+63.102466687"
Oct 30 20:20:09.923800 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:09.923777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxlb\" (UniqueName: \"kubernetes.io/projected/f575738a-d51b-460e-9ad5-e2aeb948f4bc-kube-api-access-qhxlb\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:10.039084 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.039057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:10.120324 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.120298    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:20:10.123388 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:10.123365    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df472f3_8947_48ba_a2d7_7e4ce4786505.slice/crio-86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122 WatchSource:0}: Error finding container 86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122: Status 404 returned error can't find the container with id 86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122
Oct 30 20:20:10.142452 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.142409    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6" podStartSLOduration=2.540214694 podStartE2EDuration="4.142396128s" totalImagesPullingTime="1.602181434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:10.142330716 +0000 UTC m=+63.339761754" watchObservedRunningTime="2026-10-30 20:20:10.142396128 +0000 UTC m=+63.339827163"
Oct 30 20:20:10.414426 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.414401    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c64476cfd-sv4hq"]
Oct 30 20:20:10.416612 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:10.416589    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf575738a_d51b_460e_9ad5_e2aeb948f4bc.slice/crio-b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338 WatchSource:0}: Error finding container b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338: Status 404 returned error can't find the container with id b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338
Oct 30 20:20:10.637106 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.637072    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9bdp" event={"ID":"81ab7c91-f29a-4204-b268-5407d7129914","Type":"ContainerStarted","Data":"f64c84d8444b8b7befb0b0192a024f562ee03a26839947b6a25a37b7a6b2bca9"}
Oct 30 20:20:10.637279 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.637122    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9bdp" event={"ID":"81ab7c91-f29a-4204-b268-5407d7129914","Type":"ContainerStarted","Data":"ca0f40ffd3a880ede6eaddd6861ca8250f5abdd7b762d4cc342fc2092464764e"}
Oct 30 20:20:10.638309 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.638284    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338"}
Oct 30 20:20:10.639321 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.639297    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122"}
Oct 30 20:20:10.641258 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.641236    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b" event={"ID":"a968a685-34c9-4b22-9bd6-0963e6f2dc5a","Type":"ContainerStarted","Data":"76692a573b3aa0cf0c73503898d86536a0f8c063eb69552b644f792128bb8e9f"}
Oct 30 20:20:10.641310 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.641292    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b" event={"ID":"a968a685-34c9-4b22-9bd6-0963e6f2dc5a","Type":"ContainerStarted","Data":"907505a9f223cdca65abfdf8a89e8de702e47a3355b9804eb1012869a5ec38c4"}
Oct 30 20:20:10.641350 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.641322    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b" event={"ID":"a968a685-34c9-4b22-9bd6-0963e6f2dc5a","Type":"ContainerStarted","Data":"8ed8d0e0e89d81aba3c26d76acdbd60cb8f0208cd3b04e329992e3c4f02bfc61"}
Oct 30 20:20:10.761667 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.761618    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l9bdp" podStartSLOduration=2.308622477 podStartE2EDuration="3.761605813s" totalImagesPullingTime="1.358455182s" totalInitContainerRuntime="94.528154ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:10.761189588 +0000 UTC m=+63.958620626" watchObservedRunningTime="2026-10-30 20:20:10.761605813 +0000 UTC m=+63.959036850"
Oct 30 20:20:10.880525 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:10.880471    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b" podStartSLOduration=2.124361843 podStartE2EDuration="3.880456664s" totalImagesPullingTime="1.756094821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:10.879669799 +0000 UTC m=+64.077100838" watchObservedRunningTime="2026-10-30 20:20:10.880456664 +0000 UTC m=+64.077887737"
Oct 30 20:20:11.784037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.782981    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"]
Oct 30 20:20:11.785032 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.785010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:11.838619 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.838594    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6mkkt\""
Oct 30 20:20:11.838784 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.838698    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 20:20:11.845224 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.845197    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"]
Oct 30 20:20:11.884372 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.884155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:11.884372 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.884209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6cb\" (UniqueName: \"kubernetes.io/projected/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-kube-api-access-tr6cb\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:11.985627 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.985589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:11.985815 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.985650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6cb\" (UniqueName: \"kubernetes.io/projected/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-kube-api-access-tr6cb\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:11.988486 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:11.988465    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:12.044784 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.044675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6cb\" (UniqueName: \"kubernetes.io/projected/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-kube-api-access-tr6cb\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:12.110204 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.110171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:12.289047 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.289004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:12.289219 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.289060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:12.341449 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.341385    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:20:12.341959 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.341939    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:20:12.352970 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.352943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:12.353112 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.352954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:12.389897 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.389852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:12.407045 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.407020    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:20:12.415281 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.415248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:12.441974 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.441954    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:20:12.486330 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.486302    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"]
Oct 30 20:20:12.488124 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:12.488095    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0b92cc_28ad_435b_bbcb_161bb00e2c42.slice/crio-8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af WatchSource:0}: Error finding container 8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af: Status 404 returned error can't find the container with id 8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af
Oct 30 20:20:12.502796 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.502772    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:20:12.514385 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.514362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:12.651301 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.651210    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w" event={"ID":"ff0b92cc-28ad-435b-bbcb-161bb00e2c42","Type":"ContainerStarted","Data":"8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af"}
Oct 30 20:20:12.652932 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.652904    2577 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6" exitCode=0
Oct 30 20:20:12.653076 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.652953    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"}
Oct 30 20:20:12.692372 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.692345    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:20:12.700432 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.700400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:12.762282 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.762255    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:20:12.771016 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.770989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:12.790939 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:12.790868    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f84bz"]
Oct 30 20:20:12.839133 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:12.839084    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf489b9e7_d156_4369_9afc_b38d675d1a31.slice/crio-47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca WatchSource:0}: Error finding container 47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca: Status 404 returned error can't find the container with id 47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca
Oct 30 20:20:13.096403 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.096375    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8ss9g"]
Oct 30 20:20:13.102073 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:13.102048    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0e115f_349e_4f69_b643_336b58e3c56c.slice/crio-f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536 WatchSource:0}: Error finding container f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536: Status 404 returned error can't find the container with id f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536
Oct 30 20:20:13.156047 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.156024    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tj2kt"]
Oct 30 20:20:13.157817 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:13.157797    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0184b4_b27e_4bcf_8a5f_dacb354d2574.slice/crio-29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45 WatchSource:0}: Error finding container 29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45: Status 404 returned error can't find the container with id 29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45
Oct 30 20:20:13.658328 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.658288    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f84bz" event={"ID":"f489b9e7-d156-4369-9afc-b38d675d1a31","Type":"ContainerStarted","Data":"47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca"}
Oct 30 20:20:13.659538 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.659507    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ss9g" event={"ID":"cb0e115f-349e-4f69-b643-336b58e3c56c","Type":"ContainerStarted","Data":"f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536"}
Oct 30 20:20:13.661962 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.661939    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"2e9981d054d26deccc843dea7274dbb6b7192770240a2a96eb7f23062d1bc828"}
Oct 30 20:20:13.662089 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.662006    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"bcc1efbc928699fd9deb839c119dd46714d1cd3ad578d0ae240dc89330b9d5e8"}
Oct 30 20:20:13.662089 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.662048    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"022482e6ef9795a441a6e33964b6d9fa5cf9984f1e4158587d6e8c75736684af"}
Oct 30 20:20:13.663131 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:13.663102    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tj2kt" event={"ID":"de0184b4-b27e-4bcf-8a5f-dacb354d2574","Type":"ContainerStarted","Data":"29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45"}
Oct 30 20:20:15.679186 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.679155    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ss9g" event={"ID":"cb0e115f-349e-4f69-b643-336b58e3c56c","Type":"ContainerStarted","Data":"5bd4e15f538222ab1a50bc50fa30e185d90b04581103a09e522660e49d2caf47"}
Oct 30 20:20:15.679479 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.679220    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ss9g" event={"ID":"cb0e115f-349e-4f69-b643-336b58e3c56c","Type":"ContainerStarted","Data":"a1b415556e62fb1c48def8d43e060ac62999a1e35b2445e3be6d9af61a23366a"}
Oct 30 20:20:15.681955 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.681926    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w" event={"ID":"ff0b92cc-28ad-435b-bbcb-161bb00e2c42","Type":"ContainerStarted","Data":"5891ee65df73ebd37dc39a27b865ef701887605a052c483cc9b8921b26cfeee3"}
Oct 30 20:20:15.682641 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.682627    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:15.689306 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.689124    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:15.689306 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.689164    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"dd2958150cd10d3fc4c128374b158f757f4200f5f3539d8fb2bb739169752525"}
Oct 30 20:20:15.689306 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.689212    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"22dd40d24dfea3a525821bdc3330bdef92a929008d989865e51bed0cbfee2b75"}
Oct 30 20:20:15.689306 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.689249    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" event={"ID":"f575738a-d51b-460e-9ad5-e2aeb948f4bc","Type":"ContainerStarted","Data":"6fd04fd4faedb6bb9b419bcf918b9ccefe976451121f51e4116255514956c893"}
Oct 30 20:20:15.689703 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.689628    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:15.693175 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.693138    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"}
Oct 30 20:20:15.693269 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.693202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"}
Oct 30 20:20:15.693269 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.693244    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"}
Oct 30 20:20:15.693384 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.693281    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"}
Oct 30 20:20:15.800794 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:15.800054    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8ss9g" podStartSLOduration=65.681939794 podStartE2EDuration="1m7.80004144s" totalImagesPullingTime="2.118101646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:15.798530925 +0000 UTC m=+68.995961963" watchObservedRunningTime="2026-10-30 20:20:15.80004144 +0000 UTC m=+68.997472522"
Oct 30 20:20:16.064822 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.064665    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq" podStartSLOduration=3.060570549 podStartE2EDuration="7.064649523s" totalImagesPullingTime="4.004078974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:16.064555631 +0000 UTC m=+69.261986669" watchObservedRunningTime="2026-10-30 20:20:16.064649523 +0000 UTC m=+69.262080604"
Oct 30 20:20:16.066260 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.066203    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w" podStartSLOduration=2.334491957 podStartE2EDuration="5.066185895s" totalImagesPullingTime="2.731693938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:15.922762837 +0000 UTC m=+69.120193868" watchObservedRunningTime="2026-10-30 20:20:16.066185895 +0000 UTC m=+69.263616934"
Oct 30 20:20:16.305148 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.305115    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:20:16.309352 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.309324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.362962    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tlxf9\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363054    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363118    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363158    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363237    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363269    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363340    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ophv587cgj3g\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363437    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363664    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363699    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363763    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363824    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363875    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363939    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:20:16.364037 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.363969    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:20:16.436990 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.436946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437177 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.437577 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.437534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.464211 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.464108    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:20:16.464211 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.464144    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:20:16.469920 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.469869    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:20:16.538161 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538161 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.538861 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.538624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.539402 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.539380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.540124 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.540098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.540285 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.540266    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.540652 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.540629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.540826 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.540806    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.541506 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.541482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.543772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.543117    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.544371 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.544347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.544371 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.544361    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.544641 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.544622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.546135 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.546091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.547803 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.547777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.548193 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.548168    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.548605 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.548574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.548824 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.548800    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.549159 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.549141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.549274 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.549252    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.549604 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.549580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.551029 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.550990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.602583 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.602509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.642223 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.641947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:16.707271 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.706249    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"}
Oct 30 20:20:16.707271 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.706302    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerStarted","Data":"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"}
Oct 30 20:20:16.851710 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:16.847490    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.534404863 podStartE2EDuration="7.84747166s" totalImagesPullingTime="4.296782326s" totalInitContainerRuntime="16.284471ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:16.842964798 +0000 UTC m=+70.040395834" watchObservedRunningTime="2026-10-30 20:20:16.84747166 +0000 UTC m=+70.044902696"
Oct 30 20:20:17.099641 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:17.099599    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:20:17.103303 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:17.103266    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa8d11e_1a70_4b91_8270_dd8c93d8511c.slice/crio-97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1 WatchSource:0}: Error finding container 97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1: Status 404 returned error can't find the container with id 97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1
Oct 30 20:20:17.712832 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:17.712796    2577 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524" exitCode=0
Oct 30 20:20:17.713302 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:17.713171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"}
Oct 30 20:20:17.713302 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:17.713246    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1"}
Oct 30 20:20:18.074322 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.074289    2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c79948c48-z5442"]
Oct 30 20:20:18.120350 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.120319    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5466579cfb-k26jn"]
Oct 30 20:20:18.123904 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.123878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.213158 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.211389    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5466579cfb-k26jn"]
Oct 30 20:20:18.256330 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256493 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256493 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256609 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256688 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256688 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.256810 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.256695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357481 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357481 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357481 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.357772 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.358021 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.357825    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.358516 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.358491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.358640 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.358618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.358697 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.358660    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.358832 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.358809    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.360829 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.360806    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.360954 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.360933    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.417102 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.417049    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.438797 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.438763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:18.659832 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.659727    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:18.721971 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.721930    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tj2kt" event={"ID":"de0184b4-b27e-4bcf-8a5f-dacb354d2574","Type":"ContainerStarted","Data":"a741d142f8c16ad65972a230592f736c2e36856618c5404f18fa76edde8d7555"}
Oct 30 20:20:18.722614 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.722595    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:18.820801 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.820719    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5466579cfb-k26jn"]
Oct 30 20:20:18.825183 ip-10-0-130-195 kubenswrapper[2577]: W1030 20:20:18.825151    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2eb885_9801_45fa_ba06_681db385801e.slice/crio-c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf WatchSource:0}: Error finding container c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf: Status 404 returned error can't find the container with id c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf
Oct 30 20:20:18.847476 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:18.847420    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tj2kt" podStartSLOduration=66.207316546 podStartE2EDuration="1m10.847400479s" totalImagesPullingTime="4.640083933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:18.843993228 +0000 UTC m=+72.041424265" watchObservedRunningTime="2026-10-30 20:20:18.847400479 +0000 UTC m=+72.044831514"
Oct 30 20:20:19.348317 ip-10-0-130-195 kubenswrapper[2577]: I1030 20:20:19.348234    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:20:19.348374 ip-10-0-130-195 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:20:19.357305 ip-10-0-130-195 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:20:19.357528 ip-10-0-130-195 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:20:19.357590 ip-10-0-130-195 systemd[1]: kubelet.service: Consumed 4.425s CPU time.
Oct 30 20:20:19.365305 ip-10-0-130-195 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:20:19.433226 ip-10-0-130-195 kubenswrapper[5967]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:20:19.433226 ip-10-0-130-195 kubenswrapper[5967]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:20:19.433226 ip-10-0-130-195 kubenswrapper[5967]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:20:19.433226 ip-10-0-130-195 kubenswrapper[5967]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:20:19.435332 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435240    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.435332 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435332    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.435332 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435336    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435338    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435341    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435344    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435347    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435350    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435352    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435355    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435358    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435360    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435364    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435366    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435369    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435371    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435374    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435377    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435379    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435381    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435385    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.435428 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435388    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435390    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435393    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435395    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435398    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435400    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435403    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435405    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435408    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435411    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435413    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435416    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435418    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435421    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435424    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435427    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435429    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435432    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435441    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435444    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.435903 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435447    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435449    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435452    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435454    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435457    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435460    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435462    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435465    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435467    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435470    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435472    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435475    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435478    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435480    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435483    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435485    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435488    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435490    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435494    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435497    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.436360 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435501    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435505    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435508    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435511    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435515    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435520    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435523    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435526    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435528    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435531    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435533    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435542    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435544    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435547    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435550    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435552    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435555    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435557    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435560    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.436913 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435562    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435565    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435567    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435570    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435573    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435575    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435578    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435581    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435584    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435586    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435589    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435592    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435594    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435597    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435600    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435602    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435605    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435607    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435610    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.437381 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435612    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435616    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435619    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435621    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435623    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435626    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.435634    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436406    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436413    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436416    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436418    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436421    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436424    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436426    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436429    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436432    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436435    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436438    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436441    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436443    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.437854 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436446    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436449    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436451    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436454    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436457    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436459    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436463    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436467    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436470    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436473    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436475    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436478    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436480    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436482    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436485    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436488    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436490    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436492    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436495    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.438337 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436501    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436504    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436507    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436510    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436512    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436515    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436517    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436520    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436522    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436526    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436529    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436531    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436534    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436536    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436539    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436541    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436544    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436546    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436549    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436552    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.438802 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436554    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436557    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436559    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436563    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436567    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436569    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436572    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436577    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436580    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436583    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436585    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436588    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436591    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436594    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436597    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436599    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436603    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436606    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436609    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.439274 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436611    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436614    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436617    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436619    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436622    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436624    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436626    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436629    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436632    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436634    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436637    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436639    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436642    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436644    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436647    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436650    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436652    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436654    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436657    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.439719 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436659    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436663    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436665    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436668    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436670    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436673    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436675    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436678    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436681    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436683    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436686    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436688    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436691    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436693    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436696    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.436698    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437221    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437228    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437232    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.440184 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437236    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437239    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437242    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437245    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437247    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437270    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437274    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437278    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437281    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437283    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437286    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437289    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437291    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437294    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437299    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437302    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437304    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437307    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437310    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.440626 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437313    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437315    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437318    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437321    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437324    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437327    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437330    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437334    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437337    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437341    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437343    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437346    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437349    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437351    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437354    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437357    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437360    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437363    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.441126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437365    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437368    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437371    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437374    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437376    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437379    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437381    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437384    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437386    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437390    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437393    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437395    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437398    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437400    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437403    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437406    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437408    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437411    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437415    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437418    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.441707 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437421    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437423    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437426    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437428    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437431    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437433    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437436    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437438    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437441    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437443    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437446    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437449    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437451    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437454    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437457    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437459    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437462    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437465    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437474    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437478    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.442313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437481    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437484    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437487    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437489    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437492    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437494    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437497    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437500    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437502    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437505    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437509    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437512    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437514    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437517    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437519    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437521    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437524    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437527    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437529    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.442804 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437532    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437534    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437537    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437539    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437542    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437545    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.437548    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437642    5967 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437654    5967 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437664    5967 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437669    5967 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437674    5967 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437677    5967 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437681    5967 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437685    5967 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437689    5967 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437692    5967 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437695    5967 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437699    5967 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437702    5967 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437705    5967 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437707    5967 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437710    5967 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:20:19.443248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437715    5967 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437719    5967 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437723    5967 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437726    5967 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437729    5967 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437732    5967 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437753    5967 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437758    5967 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437761    5967 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437764    5967 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437767    5967 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437770    5967 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437773    5967 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437776    5967 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437779    5967 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437782    5967 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437786    5967 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437789    5967 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437792    5967 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437795    5967 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437798    5967 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437800    5967 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437805    5967 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437808    5967 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:20:19.443932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437811    5967 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437814    5967 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437817    5967 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437823    5967 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437827    5967 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437830    5967 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437833    5967 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437836    5967 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437840    5967 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437843    5967 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437846    5967 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437849    5967 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437852    5967 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437855    5967 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437859    5967 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437862    5967 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437866    5967 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437869    5967 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437872    5967 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437875    5967 flags.go:64] FLAG: --help="false"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437878    5967 flags.go:64] FLAG: --hostname-override="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437881    5967 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437884    5967 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:20:19.444516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437887    5967 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437903    5967 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437906    5967 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437909    5967 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437912    5967 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437915    5967 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437918    5967 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437920    5967 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437923    5967 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437926    5967 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437929    5967 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437931    5967 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437936    5967 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437938    5967 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437941    5967 flags.go:64] FLAG: --lock-file=""
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437944    5967 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437946    5967 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437952    5967 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437957    5967 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437960    5967 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437966    5967 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437969    5967 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437972    5967 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437976    5967 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:20:19.445116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437979    5967 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437982    5967 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437986    5967 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437989    5967 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437993    5967 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437996    5967 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.437998    5967 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438001    5967 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438004    5967 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438007    5967 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438010    5967 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438012    5967 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438019    5967 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438022    5967 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438025    5967 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438028    5967 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438031    5967 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438033    5967 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438036    5967 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438039    5967 flags.go:64] FLAG: --port="10250"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438043    5967 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438046    5967 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-07de1ee85cd06b297"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438049    5967 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438052    5967 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438055    5967 flags.go:64] FLAG: --register-node="true"
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438059    5967 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:20:19.446058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438062    5967 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438065    5967 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438068    5967 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438071    5967 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438075    5967 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438078    5967 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438081    5967 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438084    5967 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438087    5967 flags.go:64] FLAG: --runonce="false"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438090    5967 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438093    5967 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438096    5967 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438099    5967 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438101    5967 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438104    5967 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438107    5967 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438110    5967 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438113    5967 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438115    5967 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438118    5967 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438121    5967 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438124    5967 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438127    5967 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438129    5967 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438134    5967 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:20:19.446806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438137    5967 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438142    5967 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438145    5967 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438148    5967 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438151    5967 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438154    5967 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438158    5967 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438161    5967 flags.go:64] FLAG: --v="2"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438165    5967 flags.go:64] FLAG: --version="false"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438169    5967 flags.go:64] FLAG: --vmodule=""
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438173    5967 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438177    5967 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438306    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438310    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438313    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438315    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438318    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438321    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438324    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438327    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438329    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438332    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.447436 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438334    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438337    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438340    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438342    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438345    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438348    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438350    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438353    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438356    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438358    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438360    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438366    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438369    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438373    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438377    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438380    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438385    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438388    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438390    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.448087 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438393    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438395    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438398    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438401    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438403    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438406    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438408    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438411    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438413    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438416    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438418    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438420    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438423    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438426    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438428    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438431    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438433    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438436    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438438    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438441    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.448562 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438443    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438445    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438448    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438450    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438454    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438457    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438459    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438462    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438464    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438469    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438471    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438474    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438476    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438478    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438481    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438483    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438486    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438489    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438491    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.449133 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438493    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438496    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438498    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438501    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438503    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438506    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438508    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438511    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438513    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438516    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438518    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438521    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438523    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438526    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438528    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438530    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438533    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438536    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438539    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438542    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.449726 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438544    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438547    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438550    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438553    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438555    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438558    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438560    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438562    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438565    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438567    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438570    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438572    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438575    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438578    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438580    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438583    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438585    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.450319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438587    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.438613    5967 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438673    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438677    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438680    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438683    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438686    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438688    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438691    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438693    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438696    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438699    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438701    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438705    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438708    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438711    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.450784 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438713    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438718    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438720    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438723    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438725    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438728    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438730    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438733    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438751    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438755    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438758    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438761    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438764    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438766    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438769    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438771    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438774    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438776    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438779    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438781    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.451329 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438784    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438786    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438788    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438791    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438793    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438796    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438799    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438801    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438804    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438807    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438810    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438813    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438815    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438819    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438821    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438824    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438826    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438829    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438831    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.451889 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438834    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438836    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438839    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438841    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438844    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438847    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438850    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438852    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438854    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438858    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438861    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438863    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438867    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438870    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438874    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438877    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438880    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438882    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438885    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.452388 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438887    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438890    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438893    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438897    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438900    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438902    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438905    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438908    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438910    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438913    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438916    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438918    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438921    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438923    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438926    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438928    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438931    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438936    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438939    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438941    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.453155 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438944    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438946    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438949    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438951    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438953    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438956    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438958    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438961    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438963    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438966    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438968    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438971    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438973    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.438975    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.443344    5967 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.443358    5967 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.443376    5967 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.443380    5967 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443454    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.453762 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443459    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443462    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443465    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443468    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443470    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443473    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443476    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443478    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443481    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443483    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443486    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443494    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443497    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443500    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443503    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443506    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443508    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443511    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443514    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.454349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443516    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443519    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443522    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443524    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443527    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443531    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443535    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443538    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443541    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443543    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443546    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443549    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443551    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443554    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443556    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443558    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443561    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443564    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443567    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.454825 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443569    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443572    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443574    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443577    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443579    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443582    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443590    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443592    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443595    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443597    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443600    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443602    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443604    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443607    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443609    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443612    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443615    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443617    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443620    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443623    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.455269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443625    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443629    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443633    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443636    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443639    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443642    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443645    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443648    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443650    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443653    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443656    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443658    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443661    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443663    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443666    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443669    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443671    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443673    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443676    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443683    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.455727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443686    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443689    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443691    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443693    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443696    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443698    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443701    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443703    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443706    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443708    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443711    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443714    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443716    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443719    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443721    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443725    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443728    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443732    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443755    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443760    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.456212 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443763    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443767    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443771    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443775    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443778    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443782    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.443786    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.443816    5967 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444044    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444051    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444053    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444056    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444065    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444069    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444074    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444078    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.456768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444082    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444085    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444089    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444093    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444098    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444102    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444106    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444110    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444113    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444116    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444118    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444121    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444123    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444126    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444128    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444131    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444133    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444136    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444138    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.457180 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444140    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444143    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444145    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444148    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444150    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444154    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444158    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444162    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444165    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444168    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444178    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444181    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444183    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444186    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444194    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444197    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444199    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444202    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444205    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.457644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444207    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444210    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444212    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444215    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444217    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444220    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444222    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444224    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444227    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444229    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444232    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444237    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444240    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444242    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444245    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444247    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444250    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444252    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444254    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.458299 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444257    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444259    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444262    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444264    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444266    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444274    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444277    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444279    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444282    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444284    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444287    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444289    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444292    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444294    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444299    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444301    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444304    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444306    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444309    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444311    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.458768 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444313    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444316    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444318    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444321    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444323    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444326    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444329    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444331    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444334    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444336    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444339    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444342    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444344    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444347    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444349    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444351    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444354    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444356    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444365    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444368    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.459249 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444370    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.444398    5967 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444446    5967 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444450    5967 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444452    5967 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444455    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444457    5967 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444461    5967 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444464    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444467    5967 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444469    5967 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444471    5967 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444474    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444476    5967 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444479    5967 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444481    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:20:19.459763 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444484    5967 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444487    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444489    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444491    5967 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444494    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444496    5967 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444499    5967 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444501    5967 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444504    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444507    5967 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444509    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444511    5967 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444514    5967 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444517    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444520    5967 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444527    5967 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444530    5967 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444533    5967 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444536    5967 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444538    5967 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:20:19.460145 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444541    5967 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444544    5967 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444546    5967 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444550    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444553    5967 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444555    5967 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444558    5967 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444560    5967 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444563    5967 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444565    5967 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444567    5967 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444570    5967 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444572    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444575    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444577    5967 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444579    5967 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444582    5967 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444584    5967 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444587    5967 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444589    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:20:19.460643 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444591    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444594    5967 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444596    5967 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444599    5967 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444601    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444604    5967 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444606    5967 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444609    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444618    5967 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444620    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444623    5967 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444625    5967 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444627    5967 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444630    5967 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444632    5967 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444636    5967 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444638    5967 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444641    5967 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444643    5967 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:20:19.461230 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444646    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444648    5967 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444651    5967 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444653    5967 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444656    5967 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444659    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444661    5967 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444665    5967 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444669    5967 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444673    5967 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444677    5967 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444679    5967 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444682    5967 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444685    5967 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444688    5967 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444690    5967 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444692    5967 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444695    5967 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444697    5967 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:20:19.461693 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444700    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444702    5967 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444705    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444713    5967 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444716    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444718    5967 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444721    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444723    5967 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444725    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444729    5967 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444731    5967 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444734    5967 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444762    5967 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:20:19.444766    5967 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.444838    5967 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.447228    5967 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.447311    5967 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.447564    5967 server.go:1014] "Starting client certificate rotation"
Oct 30 20:20:19.462208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.447697    5967 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.447759    5967 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:14:07 +0000 UTC" deadline="2028-07-26 18:29:04.798741748 +0000 UTC"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.447791    5967 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15238h8m45.350952773s"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.448051    5967 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.449398    5967 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.450717    5967 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.452659    5967 log.go:25] "Validated CRI v1 image API"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.453547    5967 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:20:19.462645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.455938    5967 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 91d4c5d2-ce63-4a60-8963-d45a438a0f04:/dev/nvme0n1p3 d9219cde-95c5-49f7-8807-55e9791b21e3:/dev/nvme0n1p4]
Oct 30 20:20:19.462896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.455957    5967 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45/userdata/shm major:0 minor:406 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca/userdata/shm major:0 minor:399 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af/userdata/shm major:0 minor:395 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1/userdata/shm major:0 minor:505 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf/userdata/shm major:0 minor:518 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caaebb157709cbd7c674118147873bdc6345f4a844bd05dee7dd87cd7256c469/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caaebb157709cbd7c674118147873bdc6345f4a844bd05dee7dd87cd7256c469/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~projected/kube-api-access-2vjj7:{mountpoint:/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~projected/kube-api-access-2vjj7 major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/alertmanager/9 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/config-reloader/6 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~empty-dir/config-out major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~projected/kube-api-access-7z2q9:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~projected/kube-api-access-7z2q9 major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~projected/tls-assets major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/config-volume major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/web-config major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~projected/kube-api-access-kxg45:{mountpoint:/var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~projected/kube-api-access-kxg45 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/213915b5-832d-44c7-a046-12833d574630/volumes/kubernetes.io~projected/kube-api-access-ncrsz:{mountpoint:/var/lib/kubelet/pods/213915b5-832d-44c7-a046-12833d574630/volumes/kubernetes.io~projected/kube-api-access-ncrsz major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f19916b-62f3-4a81-ab47-4cde132463c5/volumes/kubernetes.io~projected/kube-api-access-c546q:{mountpoint:/var/lib/kubelet/pods/2f19916b-62f3-4a81-ab47-4cde132463c5/volumes/kubernetes.io~projected/kube-api-access-c546q major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~projected/kube-api-access-q2z4d:{mountpoint:/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~projected/kube-api-access-q2z4d major:0 minor:517 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:516 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:511 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bded661-99cb-4a66-a36b-4e0a446321ef/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4bded661-99cb-4a66-a36b-4e0a446321ef/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~projected/kube-api-access-qgnn2:{mountpoint:/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~projected/kube-api-access-qgnn2 major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/707f544c-cf0c-4c54-bd0f-6dec8791c59f/volumes/kubernetes.io~projected/kube-api-access-wqkwp:{mountpoint:/var/lib/kubelet/pods/707f544c-cf0c-4c54-bd0f-6dec8791c59f/volumes/kubernetes.io~projected/kube-api-access-wqkwp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75417679-7317-484c-bc01-0d7870ae7825/volumes/kubernetes.io~projected/kube-api-access-6b8z2:{mountpoint:/var/lib/kubelet/pods/75417679-7317-484c-bc01-0d7870ae7825/volumes/kubernetes.io~projected/kube-api-access-6b8z2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75417679-7317-484c-bc01-0d7870ae7825/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/75417679-7317-484c-bc01-0d7870ae7825/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~projected/kube-api-access-8cv52:{mountpoint:/var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~projected/kube-api-access-8cv52 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~projected/kube-api-access-jtqzd:{mountpoint:/var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~projected/kube-api-access-jtqzd major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:499 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:495 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~projected/kube-api-access-l9mt8:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~projected/kube-api-access-l9mt8 major:0 minor:504 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~projected/tls-assets major:0 minor:497 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/config major:0 minor:502 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:503 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:490 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:498 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:501 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:494 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/web-config major:0 minor:499 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95c43767-28f8-4bfc-960c-fcf1ee6c0207/volumes/kubernetes.io~projected/kube-api-access-6rdq4:{mountpoint:/var/lib/kubelet/pods/95c43767-28f8-4bfc-960c-fcf1ee6c0207/volumes/kubernetes.io~projected/kube-api-access-6rdq4 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95c43767-28f8-4bfc-960c-fcf1ee6c0207/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/95c43767-28f8-4bfc-960c-fcf1ee6c0207/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97/volumes/kubernetes.io~projected/kube-api-access-nc44z:{mountpoint:/var/lib/kubelet/pods/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97/volumes/kubernetes.io~projected/kube-api-access-nc44z major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~projected/kube-api-access-4w25b:{mountpoint:/var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~projected/kube-api-access-4w25b major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4bac513-58af-4a06-a6e6-f419b8a9bc60/volumes/kubernetes.io~projected/kube-api-access-hjflc:{mountpoint:/var/lib/kubelet/pods/b4bac513-58af-4a06-a6e6-f419b8a9bc60/volumes/kubernetes.io~projected/kube-api-access-hjflc major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volumes/kubernetes.io~projected/kube-api-access-89vsd:{mountpoint:/var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volumes/kubernetes.io~projected/kube-api-access-89vsd major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0e115f-349e-4f69-b643-336b58e3c56c/volumes/kubernetes.io~projected/kube-api-access-gvkkh:{mountpoint:/var/lib/kubelet/pods/cb0e115f-349e-4f69-b643-336b58e3c56c/volumes/kubernetes.io~projected/kube-api-access-gvkkh major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0e115f-349e-4f69-b643-336b58e3c56c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cb0e115f-349e-4f69-b643-336b58e3c56c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9185192-48b5-4f1d-9e4c-8344f4c64058/volumes/kubernetes.io~projected/kube-api-access-fjrfg:{mountpoint:/var/lib/kubelet/pods/d9185192-48b5-4f1d-9e4c-8344f4c64058/volumes/kubernetes.io~projected/kube-api-access-fjrfg major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9185192-48b5-4f1d-9e4c-8344f4c64058/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d9185192-48b5-4f1d-9e4c-8344f4c64058/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de0184b4-b27e-4bcf-8a5f-dacb354d2574/volumes/kubernetes.io~projected/kube-api-access-bktlv:{mountpoint:/var/lib/kubelet/pods/de0184b4-b27e-4bcf-8a5f-dacb354d2574/volumes/kubernetes.io~projected/kube-api-access-bktlv major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de295ab7-6640-4744-826b-e780a3d7e597/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/de295ab7-6640-4744-826b-e780a3d7e597/volumes/kubernetes.io~secret/tls-certificates major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e97088a0-b646-48d9-818b-c932841333e4/volumes/kubernetes.io~projected/kube-api-access-jkkff:{mountpoint:/var/lib/kubelet/pods/e97088a0-b646-48d9-818b-c932841333e4/volumes/kubernetes.io~projected/kube-api-access-jkkff major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~projected/kube-api-access-6j9sm:{mountpoint:/var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~projected/kube-api-access-6j9sm major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f489b9e7-d156-4369-9afc-b38d675d1a31/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f489b9e7-d156-4369-9afc-b38d675d1a31/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~projected/kube-api-access-qhxlb:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~projected/kube-api-access-qhxlb major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff0b92cc-28ad-435b-bbcb-161bb00e2c42/volumes/kubernetes.io~projected/kube-api-access-tr6cb:{mountpoint:/var/lib/kubelet/pods/ff0b92cc-28ad-435b-bbcb-161bb00e2c42/volumes/kubernetes.io~projected/kube-api-access-tr6cb major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff0b92cc-28ad-435b-bbcb-161bb00e2c42/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/ff0b92cc-28ad-435b-bbcb-161bb00e2c42/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:393 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f2d9a4dab4092034c7aa4796f91038847265521cf0b1645102f28e4fa2c5a4c8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7e711e31d62a4917a11c9c81510066f6710eecdaed3985fe53b41374d95b434c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ddbd8290c7f40ed6d01db84b7429f2597aa0fcb35d2e469a1bb74818c1c0516e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/5561b8fcf8d0f0fbd84a428285b0a236a96b33f7f8dd783b7be9fbc3a07051e2/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0d0de225dc8ea3ab32757469497db741ff495553e71e3bc40ce1bc7b3320d7a1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/82530650b6bcaacafec6cd2f0c67cd810b8f5d45c0d540ffdee84f0f5aef0c41/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bd38dd770274eb192aebc1105a212de524cb7e4b3300c72ebdffadf0c77c26af/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0c3d6390c65c5062d150b8fe288d784f39150325bcedb2f6a3bb62d0bf9eb02c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/787e60510d39c66282c3fb147237e9ae1d8ba27015206363417ce8b921fc4225/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b43f7738b2b01d5cbda19c8a8dd15b4311c2f2d32f1ac3732f25ac1b383809c2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/fa9e2246093fce835121b9e976f681bd6fa9fdada0e226a100ef3a7c3ee9134e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/5157bda374df2267feb0143a504dfcc95f385a428d100f2f967b904147d2d23a/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3e8211df218444e3e36dd7b966c2848d94de60be15ace2a21df3c3e753838906/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6c9830502d33e57aacc2b9892e7fb8f0a3943ee8cf9ea75ba320301386eac0a1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c902ec2d236526abdc01819d95c52ce293f09fc930126aae260f2b72db4cafad/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2b49f30b12a8fab1b0cf596b5e89441c825cd8f2f76d6a79b70e80e74f4ee409/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1f18f1179e45232c5f4a9fec4c1f3b415e34b81352a712a0e38b871efb9facfb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d95f305cf3cc2c398ccf208ebdbbe95658e0666991b23243dc77388137f252c0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a17f13e951b75d368675c31e65d0f02c5032fdc3040132b5fd1e5a6556d525fd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/09605869b7c6c64305aea25579cba7742a4bdb1d7f393a4b5e6e3f59dbdf1b4c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/54c1921f18f935b82fd053916479eb68c4769c1fb02f5766810f06b5c58d7b66/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/755917979f9c765526ed1b887ee008cadeba7fb18ccb2c6d579cf3cd595478f7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8b577f0ba22b3d9179fb3e2c92b026d8ff6794d0be35e1dfdb6f12cad3f10f2f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/9c8771841a80c325e85b5ed006b2e170d66b14f6acfa74ee71fa8b1a0ca6bc41/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7bae0a7a5077e8a7209e4da6059bc29e5566f4bb084f180e6f330bf8a27e8449/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/6e3b74da8b3320dc7490c1bc4717736d439e6815afab441a55a252dd51f21ac1/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/de759e043671c6898387b610081d8260a1e1c5a365ae85ed537600ae8835c41c/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/ff15004d05a8d15fd729dae0c319af18f5883a73617feb5e3ab97a744fbd9f76/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/5289900ccacc1cec2dc44eaefb5b6e5391f094c9d35bf5f87174e1dbaaa5aa07/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/277e329bd5700b9b59aadcc324fc47adfe6afbba55b06982b51a1efac8711324/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/47a244761f9a41d45f0c53bade6ba0fed1f3e69236c44301e02d7d855a3f1faa/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3f8d9274bc0c80d7b6bf8a0d3118a932379e67d59bfe281d72c53bec448f73a0/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/698551e87618cb0ba2a013bba127986f4d10cca57c4e3938310461f16aadc4d8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/e1c04c7d57a53c5a8f590dae0c7fc1a15fa0d42b6c91a7ba8dbfb70400efea69/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/6584a7f1e49c5bd665b27d0148c55e3eecb2e2ce2ea998d0afe05cc483912eb8/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/d9b67f9480f96185ad983a9ebc9d5b590aaf664f021b427faa9f007d7869a19d/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/04a8ae6361b1af7cd44dcccf59ee53f697386e02b60502db28189f82fbd8db84/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/f565bcc69fd791ebc5a6c82f891aaf294be78cbfda92d35afc0cbb3109097045/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/e76fee2358db2c98f3a3a4e3026a081cfab983fff2c6befa4063f63536ffa985/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/41472affa8cbdd2585c1a2ad80ed82c3817204ff425600befce5ad27a4eb0ac1/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/e058221b9a9367c84a481322ab1cfe903db53db77ddbfba7dfc48ea9cef55a6e/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/ab247ac9bf06de4b91142438b86c31cd442ba059e1a9080a695be51ac4023166/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/be8fa9cd5885662ffb56493b6cb4e8860ec13bf48bc7af6c245c804a4840eed6/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/a0333b73cab6ea831db8ee6b5fbd50bf33bde35ff520f0d23db49b7266b29685/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/449dbc24742c8843ab2dfb1fd2abc1e315711447903ec58c8c75fc402e32934c/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/8917be84322ce1f373a2f2c5b5d6aa6a04199eab763dc3cbc968cddcc91ddacb/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/ee090694b860709ffa2a5ccfb94a9bd504c4def79b7b5343065e6ad7b686cf66/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/d9200b60cd640a42677af8761e6ae21b96fddf85695debb3a78a511c1bc991c6/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/b563d1c159a424771f2dba4aa75dac1294a8bee394ae190c020535fa393b6fd7/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/7cc3dbac9568177aec8bf6dd45d4fc344a6e8f983d744c3faa06dbf4cc40f834/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/89d65f2e27a060bebbfbe201b0a575b22ce10eca7b19479cf285ecf28aaf325c/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/4403cb7cd69b0ca8d8171eb8addc4366d75e0f814bfa207a2b1263a2293c9885/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/21fc75eb34799bf8fe6ec809f527c9dd3c26e603681ee86903f772c92574efad/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/2a5ce0baa9945c8048a988566032bfba5ea51ee354794a0079c1b440a1163619/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/6678463cabccb37a33ec6479d74b597ced27c7fb7e5d7b60d9797dad7033853c/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/0a7c2ac715033af87efbaddd19b36e65999ef6be15e9c8911f1b43382e1341bd/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/e875de4e0f84935369d7a83d0232dfb8e918b88e3000fcde6191b45d8e341c35/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/b32b208d09d45b41a55c52e1e7d14bc95893ff9af389027d8102af5dd54e6795/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/6f9c8053ce457b6994ed4e02599e179980fef14754e6acea7a4e3c583d623931/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/27929f94d93455ec96938fcc292a9de61427f628fccd1e3db0e820ef8a9286e5/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/71bcb3ceb34db810e383bad94f1624da38cc35ca0aae37bc3ec1f981180cd99e/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/b6abeba43904a7430f4ed9869202f6aedaa44be4f40832de06b0cd53c8f81654/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/3379bf5fb5740f60802b3779959936db519eb8abf8a5f6ed44a975f6c9378fc9/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/927082a01fe6debeca8d4f9d55f3b76aea2cd0f084047c53c8a7e255592a2161/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/155b239d67ed9671b21e7cdb692a5ab2723002ec78225c0befcb0f298263f246/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/dc928baa0d9c2001a358027b0877343a35495b8a2a6826e920410bae450e2886/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/600eabd49f635a4cda51793a8a5ff3ab99759de3e5ee8af622777de16b208e99/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/a1f29d91340706a4e266eec2e2f2954ba54f4f776f93e30ca0f2e9c1f0560d1f/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/a74bbc2f5a2b2b164f76d483fc33182f27abbdee69346d67840c427b5b30852a/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/57f397666498d078a80328ab3ab3dc35e21fb1454c5d5f4c570a3c8082b4ff35/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/5b275f206fda206948d5ed5be0c7c00237c92044c6f64a78f5fc80cd40dc614e/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/e389830b002bb7028ba7cb991071d4a98af33a46363a29e5aa59b6f90eb1a6dd/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/76247ba9956d58a6b677762467ed5912cea292cae91ac0bff4a210bd1ad6de0d/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/21bb4f9a644135e5f3234c9c1fd39e40942c50aeb084102d4317e71602a66083/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/6ab5aa35bc995d84f1a4d446fd9645c39140de1ad1f411a314de93573b9f0ec6/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/a308dbf3627b26235627a499a80c644ba5e6dba7caef5bcc7db803e91288f0da/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/754e529a8a86590d534d0b5b8256f2e3d837481b971282a75524f47b19b5ae8a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/d462915b0baa7850d09bfcda306f2382f63e8f00cbc8b8771490a15a25cc39a0/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7c98bbda5a5bfda310a868cfb20e82886810d66e097d490db868bfa3f7f6d6ba/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dedb94960a0380260a89b504716d8783f0b9320c7ebdabe7f5e445a3f376ba0b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/977797809df5f66345dbfb5d75aacc46365c13fa77af1c7374102b5b7429baae/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b86b8f8cc3caa55a4eb4d6fedf26b69f37205a40d3d88493ec0ab3d019b4abc1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6525c6bfe9bf53e6e43f105c494cdb421c07b8b7a3915180aae222a2bb55ca59/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2c33698a822a469d19295e691686da4629e66ace7ec44c2a27879e89ef21fd80/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/cd541ec33b034bdbf55949ce0d01f7a923ffd0187ef26f6cd99e4f3e390ec58d/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/17eef15ab5c28865fe4fc2a65435eaaf098ae161b1ad695f77ffa83e4d94bc1e/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 20:20:19.466650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.466158    5967 manager.go:217] Machine: {Timestamp:2026-10-30 20:20:19.465641638 +0000 UTC m=+0.066443589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3213114 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24e34c235d2b0180964f5c6f2916d1 SystemUUID:ec24e34c-235d-2b01-8096-4f5c6f2916d1 BootID:9484d53e-c2bc-4da1-945a-96140e4812b4 Filesystems:[{Device:/var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95c43767-28f8-4bfc-960c-fcf1ee6c0207/volumes/kubernetes.io~projected/kube-api-access-6rdq4 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de295ab7-6640-4744-826b-e780a3d7e597/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd94abe7099c3972dd722feaafc326ec8e1cf6195de9c9fa0159f33954779f19/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fed04d2c51e5e108dab658a624e37fe95b28f86483a5420ada1b35b5380c3af/userdata/shm DeviceMajor:0 DeviceMinor:395 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0e115f-349e-4f69-b643-336b58e3c56c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:398 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b91e7f0dbf10f152938313d9832aebbcbcfec16093b5fdf98ac65d60f6be0b78/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:325 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/75417679-7317-484c-bc01-0d7870ae7825/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba04dee5fb1e3d5b802f1758f0837d3d3fb9b2de0499360264e0e184a605b599/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:497 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff0b92cc-28ad-435b-bbcb-161bb00e2c42/volumes/kubernetes.io~projected/kube-api-access-tr6cb DeviceMajor:0 DeviceMinor:394 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:502 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:516 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~projected/kube-api-access-q2z4d DeviceMajor:0 DeviceMinor:517 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0510654b607c342b7a6cbf3ff1200161111da60163432b477d445a1092dc1fdf/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~projected/kube-api-access-8cv52 DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:385 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:490 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:500 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:503 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97/volumes/kubernetes.io~projected/kube-api-access-nc44z DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aa500db2daf4f0c17b8caf5b72588c38e8e9917be89f19ab5f01dffab1ec822/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~projected/kube-api-access-jtqzd DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:350 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:494 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:499 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d97486dd34bf36b94eeef8c740be1bce502c71f3e310ef385d4ac3901a7f326/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~projected/kube-api-access-4w25b DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c9789942-84f9-4af9-ba2d-52d3115b5882/volumes/kubernetes.io~projected/kube-api-access-89vsd DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b039ebb98c2b91484f2f49cc17b8ba4f723c8ace3c17fa63960b7018f14f8fb/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0e115f-349e-4f69-b643-336b58e3c56c/volumes/kubernetes.io~projected/kube-api-access-gvkkh DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f19916b-62f3-4a81-ab47-4cde132463c5/volumes/kubernetes.io~projected/kube-api-access-c546q DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/213915b5-832d-44c7-a046-12833d574630/volumes/kubernetes.io~projected/kube-api-access-ncrsz DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de0184b4-b27e-4bcf-8a5f-dacb354d2574/volumes/kubernetes.io~projected/kube-api-access-bktlv DeviceMajor:0 DeviceMinor:403 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0ab3d6c53e70f5f2fd71b6be6ad351f31883d21638ecd7278f5b70ebe1dc338/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5582d3013c9890c6f5620158ae399a95308c6ce772192573248525884428944e/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efa78b5cd164bd12212829a250c3cd76f7fb136df436dc6f56d46888d0d841d3/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~projected/kube-api-access-7z2q9 DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ad38a5de0058cdb4f623a6ac209040a2719a1a40a963cc628a69d250d358fb8/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29b080d4dc7454e3dcdc01974426f61a736ef63757e83c52c247587f8c89cc45/userdata/shm DeviceMajor:0 DeviceMinor:406 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/beed40dc19b220f25a55a6b73970de8c80bdc07dfdb176a137519d019f6db7c1/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a968a685-34c9-4b22-9bd6-0963e6f2dc5a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9185192-48b5-4f1d-9e4c-8344f4c64058/volumes/kubernetes.io~projected/kube-api-access-fjrfg DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea32f848b19736547b1b3020a8429d3480a20dd46a862fa9849ee8943f2156e8/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:498 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf/userdata/shm DeviceMajor:0 DeviceMinor:518 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95c43767-28f8-4bfc-960c-fcf1ee6c0207/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/707f544c-cf0c-4c54-bd0f-6dec8791c59f/volumes/kubernetes.io~projected/kube-api-access-wqkwp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~projected/kube-api-access-l9mt8 DeviceMajor:0 DeviceMinor:504 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:501 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/461fa8db3cd9e37e8e51a0ca63b6b8e234be4da236d928de446e5b566c212623/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:495 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:496 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:382 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47ca7553818236b6ecd4ab35136a7fcad3bc9216d4cb414b241f8f6557e917ca/userdata/shm DeviceMajor:0 DeviceMinor:399 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1/userdata/shm DeviceMajor:0 DeviceMinor:505 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caaebb157709cbd7c674118147873bdc6345f4a844bd05dee7dd87cd7256c469/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b460c77b6cee15488215b2b6b8ac41485394d3e8a5e87bb783836b6b9f71fc82/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:381 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~projected/kube-api-access-qhxlb DeviceMajor:0 DeviceMinor:386 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:511 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b4bac513-58af-4a06-a6e6-f419b8a9bc60/volumes/kubernetes.io~projected/kube-api-access-hjflc DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~projected/kube-api-access-kxg45 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e97088a0-b646-48d9-818b-c932841333e4/volumes/kubernetes.io~projected/kube-api-access-jkkff DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81ab7c91-f29a-4204-b268-5407d7129914/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f489b9e7-d156-4369-9afc-b38d675d1a31/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:397 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes/kubernetes.io~projected/kube-api-access-qgnn2 DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~projected/kube-api-access-2vjj7 DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bded661-99cb-4a66-a36b-4e0a446321ef/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9185192-48b5-4f1d-9e4c-8344f4c64058/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f575738a-d51b-460e-9ad5-e2aeb948f4bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:373 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1f681324-576a-4ae5-95fd-2a7e8f8ce67a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/75417679-7317-484c-bc01-0d7870ae7825/volumes/kubernetes.io~projected/kube-api-access-6b8z2 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eadb37fe758caf7561467bc55b8807ee75e19fa60e6529d5274129048213cc90/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79d81e86-e9cb-4bf1-9286-74af62e818ee/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec59824c-8bdf-4099-9124-26a29ae8e876/volumes/kubernetes.io~projected/kube-api-access-6j9sm DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff0b92cc-28ad-435b-bbcb-161bb00e2c42/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:393 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f206f5666e33b68d8153705dd40d67a357bd832dbe8b941481238b30b6daa536/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:499 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0510654b607c342 MacAddress:ca:33:56:ea:df:54 Speed:10000 Mtu:8901} {Name:29b080d4dc7454e MacAddress:8e:12:c9:42:ea:86 Speed:10000 Mtu:8901} {Name:47ca7553818236b MacAddress:ee:e0:cb:03:7d:fd Speed:10000 Mtu:8901} {Name:71fc723cf58149f MacAddress:02:38:b4:92:45:c5 Speed:10000 Mtu:8901} {Name:7d97486dd34bf36 MacAddress:56:ec:31:ec:a5:11 Speed:10000 Mtu:8901} {Name:86a304bb9a84088 MacAddress:02:bc:f6:25:cc:25 Speed:10000 Mtu:8901} {Name:8fed04d2c51e5e1 MacAddress:26:18:01:15:16:1f Speed:10000 Mtu:8901} {Name:97ba26ab7e90948 MacAddress:de:29:8e:5d:1a:78 Speed:10000 Mtu:8901} {Name:b0ab3d6c53e70f5 MacAddress:12:41:f9:20:33:78 Speed:10000 Mtu:8901} {Name:b460c77b6cee154 MacAddress:26:4e:9b:e6:71:75 Speed:10000 Mtu:8901} {Name:ba04dee5fb1e3d5 MacAddress:72:84:99:cb:1f:67 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3f:a5:df:78:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:0c:ae:67:c1:83 Speed:0 Mtu:8901} {Name:c57d7fa9bb0c83d MacAddress:56:c1:bf:be:85:e4 Speed:10000 Mtu:8901} {Name:caaebb157709cbd MacAddress:8e:4c:c4:e2:5d:03 Speed:10000 Mtu:8901} {Name:cd94abe7099c397 MacAddress:56:0e:e4:11:46:93 Speed:10000 Mtu:8901} {Name:da8fecc3894f9db MacAddress:ba:ef:e5:85:56:66 Speed:10000 Mtu:8901} {Name:eadb37fe758caf7 MacAddress:be:88:0c:a9:8f:d6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3f:a5:df:78:81 Speed:0 Mtu:9001} {Name:f206f5666e33b68 MacAddress:ba:0f:f6:69:c4:cb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:9e:5f:22:a2:fe:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:11:9c:e6:71:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:20:19.466650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.466651    5967 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:20:19.467064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.466702    5967 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:20:19.467289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467242    5967 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:20:19.467700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467294    5967 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-195.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:20:19.467816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467703    5967 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:20:19.467816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467716    5967 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:20:19.467816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467733    5967 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:20:19.467816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467789    5967 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:20:19.467995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.467931    5967 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:20:19.468400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468381    5967 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:20:19.468722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468710    5967 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:20:19.468822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468779    5967 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:20:19.468822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468816    5967 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:20:19.468892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468830    5967 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:20:19.468892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468842    5967 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:20:19.468892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.468853    5967 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:20:19.469815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.469798    5967 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:20:19.469815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.469811    5967 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:20:19.470423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.470406    5967 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:20:19.490521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.490496    5967 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:20:19.490521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.490515    5967 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:20:19.490702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.490635    5967 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:20:19.490913 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.490901    5967 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:20:19.490949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.490932    5967 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:20:19.491066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491055    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491075    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491083    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491090    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491096    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491102    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491107    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:20:19.491109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491112    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:20:19.491324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491120    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:20:19.491324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491126    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:20:19.491324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491140    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:20:19.491324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491149    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:20:19.491324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491176    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:20:19.491324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.491182    5967 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:20:19.494750 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.494721    5967 server.go:1263] "Started kubelet"
Oct 30 20:20:19.494853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.494808    5967 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:20:19.494963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.494894    5967 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:20:19.495137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.495048    5967 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:20:19.495371 ip-10-0-130-195 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:20:19.495532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.495469    5967 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:20:19.496178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.496156    5967 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.502307    5967 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.502338    5967 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.502415    5967 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.502423    5967 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.502442    5967 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-195.us-west-1.compute.internal\" not found"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.502558    5967 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503063    5967 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:14:08 +0000 UTC" deadline="2028-08-07 22:47:04.331651364 +0000 UTC"
Oct 30 20:20:19.503113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503087    5967 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15530h26m44.828567856s"
Oct 30 20:20:19.503553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503329    5967 factory.go:55] Registering systemd factory
Oct 30 20:20:19.503553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503344    5967 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:20:19.503553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503544    5967 factory.go:153] Registering CRI-O factory
Oct 30 20:20:19.503553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503554    5967 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:20:19.503773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503595    5967 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:20:19.503773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503613    5967 factory.go:103] Registering Raw factory
Oct 30 20:20:19.503773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.503627    5967 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.504021    5967 manager.go:319] Starting recovery of all containers
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.506941    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507011    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d81e86-e9cb-4bf1-9286-74af62e818ee" volumeName="kubernetes.io/configmap/79d81e86-e9cb-4bf1-9286-74af62e818ee-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507025    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d81e86-e9cb-4bf1-9286-74af62e818ee" volumeName="kubernetes.io/projected/79d81e86-e9cb-4bf1-9286-74af62e818ee-kube-api-access-8cv52" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507042    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ab7c91-f29a-4204-b268-5407d7129914" volumeName="kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507055    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507067    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/projected/f575738a-d51b-460e-9ad5-e2aeb948f4bc-kube-api-access-qhxlb" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507078    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507090    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75417679-7317-484c-bc01-0d7870ae7825" volumeName="kubernetes.io/secret/75417679-7317-484c-bc01-0d7870ae7825-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507104    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507116    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9789942-84f9-4af9-ba2d-52d3115b5882" volumeName="kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-env-overrides" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507127    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0e115f-349e-4f69-b643-336b58e3c56c" volumeName="kubernetes.io/projected/cb0e115f-349e-4f69-b643-336b58e3c56c-kube-api-access-gvkkh" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507140    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e97088a0-b646-48d9-818b-c932841333e4" volumeName="kubernetes.io/projected/e97088a0-b646-48d9-818b-c932841333e4-kube-api-access-jkkff" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507152    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507168    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507182    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507196    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507211    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97" volumeName="kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507221    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec59824c-8bdf-4099-9124-26a29ae8e876" volumeName="kubernetes.io/configmap/ec59824c-8bdf-4099-9124-26a29ae8e876-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507234    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507247    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9789942-84f9-4af9-ba2d-52d3115b5882" volumeName="kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507254    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0e115f-349e-4f69-b643-336b58e3c56c" volumeName="kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507261    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507269    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec59824c-8bdf-4099-9124-26a29ae8e876" volumeName="kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507278    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507284    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ab7c91-f29a-4204-b268-5407d7129914" volumeName="kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507292    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9789942-84f9-4af9-ba2d-52d3115b5882" volumeName="kubernetes.io/secret/c9789942-84f9-4af9-ba2d-52d3115b5882-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507313    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507327    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507337    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968a685-34c9-4b22-9bd6-0963e6f2dc5a" volumeName="kubernetes.io/projected/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-api-access-4w25b" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507357    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0b92cc-28ad-435b-bbcb-161bb00e2c42" volumeName="kubernetes.io/projected/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-kube-api-access-tr6cb" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507364    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507373    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507382    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95c43767-28f8-4bfc-960c-fcf1ee6c0207" volumeName="kubernetes.io/configmap/95c43767-28f8-4bfc-960c-fcf1ee6c0207-config-volume" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507389    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4bac513-58af-4a06-a6e6-f419b8a9bc60" volumeName="kubernetes.io/projected/b4bac513-58af-4a06-a6e6-f419b8a9bc60-kube-api-access-hjflc" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507396    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507405    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f19916b-62f3-4a81-ab47-4cde132463c5" volumeName="kubernetes.io/projected/2f19916b-62f3-4a81-ab47-4cde132463c5-kube-api-access-c546q" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507414    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507423    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507430    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507436    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968a685-34c9-4b22-9bd6-0963e6f2dc5a" volumeName="kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507443    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507451    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507458    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507467    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968a685-34c9-4b22-9bd6-0963e6f2dc5a" volumeName="kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507474    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968a685-34c9-4b22-9bd6-0963e6f2dc5a" volumeName="kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507483    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0b92cc-28ad-435b-bbcb-161bb00e2c42" volumeName="kubernetes.io/secret/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507490    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507498    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507505    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95c43767-28f8-4bfc-960c-fcf1ee6c0207" volumeName="kubernetes.io/projected/95c43767-28f8-4bfc-960c-fcf1ee6c0207-kube-api-access-6rdq4" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507513    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507520    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97" volumeName="kubernetes.io/projected/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-kube-api-access-nc44z" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507527    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec59824c-8bdf-4099-9124-26a29ae8e876" volumeName="kubernetes.io/projected/ec59824c-8bdf-4099-9124-26a29ae8e876-kube-api-access-6j9sm" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507535    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="213915b5-832d-44c7-a046-12833d574630" volumeName="kubernetes.io/projected/213915b5-832d-44c7-a046-12833d574630-kube-api-access-ncrsz" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507542    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f19916b-62f3-4a81-ab47-4cde132463c5" volumeName="kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507550    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507557    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9185192-48b5-4f1d-9e4c-8344f4c64058" volumeName="kubernetes.io/projected/d9185192-48b5-4f1d-9e4c-8344f4c64058-kube-api-access-fjrfg" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507571    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507579    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507588    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507596    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507603    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f489b9e7-d156-4369-9afc-b38d675d1a31" volumeName="kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507610    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d81e86-e9cb-4bf1-9286-74af62e818ee" volumeName="kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507618    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97" volumeName="kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507625    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507634    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507642    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507649    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ab7c91-f29a-4204-b268-5407d7129914" volumeName="kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507657    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507664    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ab7c91-f29a-4204-b268-5407d7129914" volumeName="kubernetes.io/projected/81ab7c91-f29a-4204-b268-5407d7129914-kube-api-access-jtqzd" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507672    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968a685-34c9-4b22-9bd6-0963e6f2dc5a" volumeName="kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507679    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9789942-84f9-4af9-ba2d-52d3115b5882" volumeName="kubernetes.io/projected/c9789942-84f9-4af9-ba2d-52d3115b5882-kube-api-access-89vsd" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507685    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-grpc-tls" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507692    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ab7c91-f29a-4204-b268-5407d7129914" volumeName="kubernetes.io/empty-dir/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-textfile" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507699    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4bac513-58af-4a06-a6e6-f419b8a9bc60" volumeName="kubernetes.io/empty-dir/b4bac513-58af-4a06-a6e6-f419b8a9bc60-tmp-dir" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507705    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9185192-48b5-4f1d-9e4c-8344f4c64058" volumeName="kubernetes.io/secret/d9185192-48b5-4f1d-9e4c-8344f4c64058-cert" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507712    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de295ab7-6640-4744-826b-e780a3d7e597" volumeName="kubernetes.io/secret/de295ab7-6640-4744-826b-e780a3d7e597-tls-certificates" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507720    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507730    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507755    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f19916b-62f3-4a81-ab47-4cde132463c5" volumeName="kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507769    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507780    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f681324-576a-4ae5-95fd-2a7e8f8ce67a" volumeName="kubernetes.io/projected/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-kube-api-access-kxg45" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507790    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75417679-7317-484c-bc01-0d7870ae7825" volumeName="kubernetes.io/projected/75417679-7317-484c-bc01-0d7870ae7825-kube-api-access-6b8z2" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507801    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95c43767-28f8-4bfc-960c-fcf1ee6c0207" volumeName="kubernetes.io/empty-dir/95c43767-28f8-4bfc-960c-fcf1ee6c0207-tmp-dir" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507812    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="707f544c-cf0c-4c54-bd0f-6dec8791c59f" volumeName="kubernetes.io/configmap/707f544c-cf0c-4c54-bd0f-6dec8791c59f-serviceca" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507825    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f681324-576a-4ae5-95fd-2a7e8f8ce67a" volumeName="kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-tuned" seLinuxMountContext=""
Oct 30 20:20:19.507784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507837    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507847    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75417679-7317-484c-bc01-0d7870ae7825" volumeName="kubernetes.io/empty-dir/75417679-7317-484c-bc01-0d7870ae7825-data-volume" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507858    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507870    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507880    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d81e86-e9cb-4bf1-9286-74af62e818ee" volumeName="kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507890    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507901    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507911    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507922    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507932    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bded661-99cb-4a66-a36b-4e0a446321ef" volumeName="kubernetes.io/secret/4bded661-99cb-4a66-a36b-4e0a446321ef-agent-certs" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507944    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9789942-84f9-4af9-ba2d-52d3115b5882" volumeName="kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-config" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507955    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="213915b5-832d-44c7-a046-12833d574630" volumeName="kubernetes.io/configmap/213915b5-832d-44c7-a046-12833d574630-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507967    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75417679-7317-484c-bc01-0d7870ae7825" volumeName="kubernetes.io/configmap/75417679-7317-484c-bc01-0d7870ae7825-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507978    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.507988    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95c43767-28f8-4bfc-960c-fcf1ee6c0207" volumeName="kubernetes.io/secret/95c43767-28f8-4bfc-960c-fcf1ee6c0207-metrics-tls" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508001    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508011    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508023    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508035    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508046    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508056    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508067    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de0184b4-b27e-4bcf-8a5f-dacb354d2574" volumeName="kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508079    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508091    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df472f3-8947-48ba-a2d7-7e4ce4786505" volumeName="kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508105    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968a685-34c9-4b22-9bd6-0963e6f2dc5a" volumeName="kubernetes.io/empty-dir/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-volume-directive-shadow" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508116    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508129    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f681324-576a-4ae5-95fd-2a7e8f8ce67a" volumeName="kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-tmp" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508162    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508173    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/configmap/f575738a-d51b-460e-9ad5-e2aeb948f4bc-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508184    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ef5c28-b16d-425a-b173-0ec514a3c3de" volumeName="kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508195    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="707f544c-cf0c-4c54-bd0f-6dec8791c59f" volumeName="kubernetes.io/projected/707f544c-cf0c-4c54-bd0f-6dec8791c59f-kube-api-access-wqkwp" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508206    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508216    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="020e1e00-032c-4533-ba58-97b7f4971d37" volumeName="kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508228    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bded661-99cb-4a66-a36b-4e0a446321ef" volumeName="kubernetes.io/configmap/4bded661-99cb-4a66-a36b-4e0a446321ef-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508239    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508249    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" volumeName="kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508262    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f19916b-62f3-4a81-ab47-4cde132463c5" volumeName="kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508273    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ab7c91-f29a-4204-b268-5407d7129914" volumeName="kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-tls" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508284    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec59824c-8bdf-4099-9124-26a29ae8e876" volumeName="kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508295    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2eb885-9801-45fa-ba06-681db385801e" volumeName="kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508306    5967 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f575738a-d51b-460e-9ad5-e2aeb948f4bc" volumeName="kubernetes.io/empty-dir/f575738a-d51b-460e-9ad5-e2aeb948f4bc-tmp" seLinuxMountContext=""
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508318    5967 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:20:19.511392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.508325    5967 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:20:19.533180 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.533126    5967 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:20:19.534665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.534641    5967 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:20:19.534665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.534663    5967 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:20:19.534875 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.534693    5967 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:20:19.534875 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.534725    5967 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:20:19.543812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.543764    5967 generic.go:356] "Generic (PLEG): container finished" podID="81ab7c91-f29a-4204-b268-5407d7129914" containerID="34e81a7227ba332d4855df2601cda81a275194f1b0b649fa83493c4c22c2f49e" exitCode=0
Oct 30 20:20:19.552335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.552303    5967 generic.go:356] "Generic (PLEG): container finished" podID="5151f952d069614d161163f3dcf066f0" containerID="5c06d9929778f39b0ad428c042cbe110a5ffec959b720b888410f4cc155a5f21" exitCode=0
Oct 30 20:20:19.571168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.571135    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="bf781a741af3599d8334f700f15cee9dc0936e239588f21d984990646b6c5ee2" exitCode=0
Oct 30 20:20:19.571168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.571161    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="48770a6fe7de3f9864b4505c69799be75fb14705c75fa97c9cde8571ee4fc197" exitCode=0
Oct 30 20:20:19.571168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.571167    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="26538b2c1fa4e37c90567af263c5a4bdf1c9ec35e0c7f95ebc8f9ea927f49c31" exitCode=0
Oct 30 20:20:19.571168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.571173    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="fe904096cdc3de4e910e5139b02af381656646c68855cce2389d462ee991ebe8" exitCode=0
Oct 30 20:20:19.571467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.571180    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="1476627a0f899533b99dadd2d105a91cb330bea6b2309e955d4a46973106e4c3" exitCode=0
Oct 30 20:20:19.571467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.571186    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f19916b-62f3-4a81-ab47-4cde132463c5" containerID="d0342510f3dcdad0c8a46ef504b0e8f8084e154187d8a6d021ced38ad08db77c" exitCode=0
Oct 30 20:20:19.580269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.580243    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6" exitCode=0
Oct 30 20:20:19.581923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.581884    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524" exitCode=0
Oct 30 20:20:19.602552 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.602489    5967 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-195.us-west-1.compute.internal\" not found"
Oct 30 20:20:19.631755 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.631712    5967 manager.go:324] Recovery completed
Oct 30 20:20:19.634848 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.634829    5967 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 20:20:19.653715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.653695    5967 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:20:19.655825 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.655809    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:20:19.655896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.655846    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:20:19.655896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.655857    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:20:19.658956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.658940    5967 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:20:19.658956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.658953    5967 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:20:19.659048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.658947    5967 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:20:19.659048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.658969    5967 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:20:19.659133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659116    5967 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:20:19.659164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659135    5967 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:20:19.659164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659157    5967 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:20:19.659164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659164    5967 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:20:19.659247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659169    5967 policy_none.go:50] "Start"
Oct 30 20:20:19.659247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659179    5967 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:20:19.659247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659190    5967 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:20:19.659344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659334    5967 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:20:19.659372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659345    5967 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:20:19.659372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659351    5967 policy_none.go:44] "Start"
Oct 30 20:20:19.659941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659927    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:20:19.659992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659956    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:20:19.659992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.659969    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:20:19.664192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.664165    5967 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:20:19.666689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.666667    5967 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:20:19.666802 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.666701    5967 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:20:19.666802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.666714    5967 server.go:87] "Starting device plugin registration server"
Oct 30 20:20:19.667033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667019    5967 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:20:19.667096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667035    5967 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:20:19.667142 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667114    5967 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:20:19.667262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667220    5967 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:20:19.667323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667275    5967 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:20:19.667323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667285    5967 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:20:19.668070 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.667962    5967 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:20:19.667261478Z","UUID":"6a2f509f-4c12-4316-92e5-229c73d3b36f","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:20:19.669427 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.669402    5967 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:20:19.669521 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.669482    5967 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-195.us-west-1.compute.internal\" not found"
Oct 30 20:20:19.669633 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.669618    5967 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:20:19.670072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.670055    5967 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:20:19.676967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.676948    5967 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:20:19.677084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.677011    5967 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:20:19.677227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.677212    5967 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:20:19.678161 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.678143    5967 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:20:19.768006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.767977    5967 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:20:19.769182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.769160    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:20:19.769357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.769195    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:20:19.769357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.769207    5967 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-195.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:20:19.769357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.769256    5967 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.775981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.775955    5967 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:20:19.835393 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.835338    5967 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"]
Oct 30 20:20:19.836197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836174    5967 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.836351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836198    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"5151f952d069614d161163f3dcf066f0","Type":"ContainerStarted","Data":"3ec51756baf52f67867d9d6798c389bf98396728dc1b324c6edc42f41dc82666"}
Oct 30 20:20:19.836351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836242    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"5151f952d069614d161163f3dcf066f0","Type":"ContainerDied","Data":"5c06d9929778f39b0ad428c042cbe110a5ffec959b720b888410f4cc155a5f21"}
Oct 30 20:20:19.836351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836262    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"5151f952d069614d161163f3dcf066f0","Type":"ContainerStarted","Data":"9f29ab6c296c3b3375a93341c61211d51f665c3e84f727bd39876207cd94adf8"}
Oct 30 20:20:19.836351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836278    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"066c09478c1923ae07faa25b9022b5b3","Type":"ContainerStarted","Data":"0c016980246d4779bae81578dc94ceac9fab7ea99c6c633803b39cc04af1395d"}
Oct 30 20:20:19.836351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836296    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal" event={"ID":"066c09478c1923ae07faa25b9022b5b3","Type":"ContainerStarted","Data":"fb2e2b4cae9760b2d9cca601630199077a2f19fa84240c763822560e0a664a55"}
Oct 30 20:20:19.836636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.836446    5967 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.877034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.876959    5967 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.877162 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.877037    5967 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.903451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.903428    5967 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:20:19.903573 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.903472    5967 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.904905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.904891    5967 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:20:19.904998 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:19.904921    5967 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.978975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.978946    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.979135 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.978981    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:19.979135 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:19.979011    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/066c09478c1923ae07faa25b9022b5b3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"066c09478c1923ae07faa25b9022b5b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.080516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.080482    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.080695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.080604    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.080695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.080614    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.080695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.080627    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/066c09478c1923ae07faa25b9022b5b3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"066c09478c1923ae07faa25b9022b5b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.080695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.080685    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5151f952d069614d161163f3dcf066f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"5151f952d069614d161163f3dcf066f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.080951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.080692    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/066c09478c1923ae07faa25b9022b5b3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" (UID: \"066c09478c1923ae07faa25b9022b5b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.469759 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.469710    5967 apiserver.go:52] "Watching apiserver"
Oct 30 20:20:20.575397 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.575369    5967 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:20:20.576113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.576091    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9zjw5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p","openshift-cluster-node-tuning-operator/tuned-jw4mv","openshift-image-registry/node-ca-qtnkw","openshift-dns/dns-default-b42wp","openshift-network-operator/iptables-alerter-dgvmc","openshift-console/console-5466579cfb-k26jn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal","openshift-multus/multus-cfstj","openshift-multus/multus-additional-cni-plugins-rk542","openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6","openshift-multus/network-metrics-daemon-8ss9g","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg","openshift-monitoring/node-exporter-l9bdp","kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal","openshift-dns/node-resolver-blv5n","kube-system/konnectivity-agent-pxcqk","openshift-monitoring/alertmanager-main-0","openshift-console/console-c79948c48-z5442","openshift-ingress-canary/ingress-canary-x9nxm","openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w","openshift-monitoring/thanos-querier-c64476cfd-sv4hq","kube-system/global-pull-secret-syncer-f84bz","openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b","openshift-monitoring/prometheus-operator-9cd89b74c-f424d","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-tj2kt","openshift-console/console-76f877b658-k6n9x","openshift-ovn-kubernetes/ovnkube-node-7pz42"]
Oct 30 20:20:20.606159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.606130    5967 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:20:20.616335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.616313    5967 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.616469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.616404    5967 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.632531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632506    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.632667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632570    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.632667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632633    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.632822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632722    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:20:20.632822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632755    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:20:20.632822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632793    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632852    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632853    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632853    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632880    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632893    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632905    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632875    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632923    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.632971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632885    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:20:20.633224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.632999    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:20:20.633224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.633006    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:20:20.633224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.633011    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.633224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.633147    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.635208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635191    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:20:20.635258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635244    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.635295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635277    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:20:20.635295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635278    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.635406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635395    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:20:20.635488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635478    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.635520 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.635502    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.670072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.670035    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q6sjr\""
Oct 30 20:20:20.670349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.670226    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bp5wk\""
Oct 30 20:20:20.670349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.670288    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:20:20.670675 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.670652    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z59pm\""
Oct 30 20:20:20.671006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.670990    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:20:20.671141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671108    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:20:20.671141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671123    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4ct2\""
Oct 30 20:20:20.671236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671172    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:20:20.671280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671269    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 20:20:20.671400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671384    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vn2fz\""
Oct 30 20:20:20.671527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671510    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:20:20.671580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671561    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:20:20.671652 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671633    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwff\""
Oct 30 20:20:20.671727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671715    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kv88f\""
Oct 30 20:20:20.671842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671825    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbmn9\""
Oct 30 20:20:20.671887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671827    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5x4ch\""
Oct 30 20:20:20.671887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671870    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:20:20.671973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671930    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:20:20.671973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.671932    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwdx\""
Oct 30 20:20:20.672055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.672029    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6wxmv\""
Oct 30 20:20:20.672194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.672179    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:20:20.672273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.672260    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lm5vf\""
Oct 30 20:20:20.672338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.672327    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:20:20.672410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.672398    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k74qc\""
Oct 30 20:20:20.672776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.672733    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:20:20.683447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683412    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.683542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683456    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.683542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683473    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:20:20.683542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683483    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-env-overrides\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.683542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683495    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:20:20.683542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683509    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683563    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-var-lib-kubelet\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683591    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-tuned\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683618    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9sm\" (UniqueName: \"kubernetes.io/projected/ec59824c-8bdf-4099-9124-26a29ae8e876-kube-api-access-6j9sm\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683645    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683668    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683687    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-tuned\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683693    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysconfig\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.683721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683720    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683770    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683721    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-env-overrides\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683795    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683835    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-conf-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683866    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-daemon-config\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683888    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683920    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683939    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683949    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.683945    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684025    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684052    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684070    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-daemon-config\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684081    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkwp\" (UniqueName: \"kubernetes.io/projected/707f544c-cf0c-4c54-bd0f-6dec8791c59f-kube-api-access-wqkwp\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.684108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684106    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6cb\" (UniqueName: \"kubernetes.io/projected/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-kube-api-access-tr6cb\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684137    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-multus\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684163    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-systemd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684212    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vsd\" (UniqueName: \"kubernetes.io/projected/c9789942-84f9-4af9-ba2d-52d3115b5882-kube-api-access-89vsd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684232    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684248    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-systemd-units\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684269    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-ovn\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684352    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-netd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684385    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684417    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684447    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213915b5-832d-44c7-a046-12833d574630-iptables-alerter-script\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684506    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684527    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-modprobe-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684542    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-conf\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684557    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.684577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684572    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684595    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684599    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213915b5-832d-44c7-a046-12833d574630-iptables-alerter-script\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684637    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684669    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684699    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684727    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684782    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bded661-99cb-4a66-a36b-4e0a446321ef-konnectivity-ca\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684815    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684812    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684843    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-sys-fs\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684912    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684930    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-system-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.684977    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-multus-certs\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685008    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685014    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bded661-99cb-4a66-a36b-4e0a446321ef-konnectivity-ca\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685034    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkff\" (UniqueName: \"kubernetes.io/projected/e97088a0-b646-48d9-818b-c932841333e4-kube-api-access-jkkff\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685051    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-host\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685069    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9185192-48b5-4f1d-9e4c-8344f4c64058-cert\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685092    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685187    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-kubelet\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.685219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685220    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-hostroot\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685251    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8z2\" (UniqueName: \"kubernetes.io/projected/75417679-7317-484c-bc01-0d7870ae7825-kube-api-access-6b8z2\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685284    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9185192-48b5-4f1d-9e4c-8344f4c64058-cert\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685286    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685332    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-cnibin\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685369    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75417679-7317-484c-bc01-0d7870ae7825-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685391    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-kubelet\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685394    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f489b9e7-d156-4369-9afc-b38d675d1a31-original-pull-secret\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685441    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9789942-84f9-4af9-ba2d-52d3115b5882-ovn-node-metrics-cert\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685479    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-socket-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685575    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c43767-28f8-4bfc-960c-fcf1ee6c0207-tmp-dir\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685609    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685639    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685661    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9789942-84f9-4af9-ba2d-52d3115b5882-ovn-node-metrics-cert\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685667    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrfg\" (UniqueName: \"kubernetes.io/projected/d9185192-48b5-4f1d-9e4c-8344f4c64058-kube-api-access-fjrfg\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685678    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c43767-28f8-4bfc-960c-fcf1ee6c0207-tmp-dir\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685692    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685717    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685769    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-bin\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685787    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-registration-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685824    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-tmp\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685843    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-sys\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685869    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685891    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-config\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685911    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-tmp\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685919    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685939    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.685955    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bac513-58af-4a06-a6e6-f419b8a9bc60-tmp-dir\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686009    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bac513-58af-4a06-a6e6-f419b8a9bc60-tmp-dir\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686026    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-textfile\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.686037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686045    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-kubelet-config\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686070    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-config\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686071    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c546q\" (UniqueName: \"kubernetes.io/projected/2f19916b-62f3-4a81-ab47-4cde132463c5-kube-api-access-c546q\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686085    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-textfile\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686103    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686121    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686141    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686156    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686171    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686188    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-var-lib-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686213    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686241    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686263    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686280    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f575738a-d51b-460e-9ad5-e2aeb948f4bc-metrics-client-ca\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686303    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrsz\" (UniqueName: \"kubernetes.io/projected/213915b5-832d-44c7-a046-12833d574630-kube-api-access-ncrsz\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686325    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bded661-99cb-4a66-a36b-4e0a446321ef-agent-certs\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686341    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686350    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686357    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-slash\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686382    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686411    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686491    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec59824c-8bdf-4099-9124-26a29ae8e876-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686514    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-wtmp\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686533    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686561    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc44z\" (UniqueName: \"kubernetes.io/projected/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-kube-api-access-nc44z\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686588    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w25b\" (UniqueName: \"kubernetes.io/projected/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-api-access-4w25b\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686615    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-etc-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686582    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bded661-99cb-4a66-a36b-4e0a446321ef-agent-certs\") pod \"konnectivity-agent-pxcqk\" (UID: \"4bded661-99cb-4a66-a36b-4e0a446321ef\") " pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686639    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-etc-kubernetes\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686678    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686706    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686733    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686758    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686794    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-sys\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686813    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686837    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686859    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-dbus\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686876    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686893    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdq4\" (UniqueName: \"kubernetes.io/projected/95c43767-28f8-4bfc-960c-fcf1ee6c0207-kube-api-access-6rdq4\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.686978    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cv52\" (UniqueName: \"kubernetes.io/projected/79d81e86-e9cb-4bf1-9286-74af62e818ee-kube-api-access-8cv52\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687009    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687035    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687156    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxlb\" (UniqueName: \"kubernetes.io/projected/f575738a-d51b-460e-9ad5-e2aeb948f4bc-kube-api-access-qhxlb\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687182    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqzd\" (UniqueName: \"kubernetes.io/projected/81ab7c91-f29a-4204-b268-5407d7129914-kube-api-access-jtqzd\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687199    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cnibin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.687203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687225    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687259    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75417679-7317-484c-bc01-0d7870ae7825-data-volume\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687313    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687351    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-systemd\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687364    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75417679-7317-484c-bc01-0d7870ae7825-data-volume\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687370    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxg45\" (UniqueName: \"kubernetes.io/projected/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-kube-api-access-kxg45\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687402    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687420    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-tls\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687437    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-k8s-cni-cncf-io\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687454    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687477    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75417679-7317-484c-bc01-0d7870ae7825-crio-socket\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687483    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687504    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-node-log\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687526    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/707f544c-cf0c-4c54-bd0f-6dec8791c59f-serviceca\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687559    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-netns\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687616    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687642    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687659    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687661    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687684    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687700    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687716    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-system-cni-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687767    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-device-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687800    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f575738a-d51b-460e-9ad5-e2aeb948f4bc-tmp\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687847    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213915b5-832d-44c7-a046-12833d574630-host-slash\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687888    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-run\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687888    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f575738a-d51b-460e-9ad5-e2aeb948f4bc-tmp\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687942    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkkh\" (UniqueName: \"kubernetes.io/projected/cb0e115f-349e-4f69-b643-336b58e3c56c-kube-api-access-gvkkh\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.687977    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75417679-7317-484c-bc01-0d7870ae7825-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688008    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688035    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688064    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688141    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688150    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688179    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688196    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688196    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/707f544c-cf0c-4c54-bd0f-6dec8791c59f-serviceca\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688216    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-root\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688155    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75417679-7317-484c-bc01-0d7870ae7825-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688290    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-bin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688318    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-netns\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688337    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95c43767-28f8-4bfc-960c-fcf1ee6c0207-config-volume\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688355    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688374    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0e115f-349e-4f69-b643-336b58e3c56c-metrics-certs\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688382    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688423    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688453    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688491    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4bac513-58af-4a06-a6e6-f419b8a9bc60-hosts-file\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688512    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-metrics-client-ca\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688530    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688548    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688584    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/707f544c-cf0c-4c54-bd0f-6dec8791c59f-host\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688602    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688619    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-grpc-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688663    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688711    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688734    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-kubernetes\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.688797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688795    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688859    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688885    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688902    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-log-socket\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688918    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.688964    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689023    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c43767-28f8-4bfc-960c-fcf1ee6c0207-metrics-tls\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689056    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-lib-modules\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689123    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f19916b-62f3-4a81-ab47-4cde132463c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689131    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjflc\" (UniqueName: \"kubernetes.io/projected/b4bac513-58af-4a06-a6e6-f419b8a9bc60-kube-api-access-hjflc\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689203    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cni-binary-copy\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689235    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689248    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c43767-28f8-4bfc-960c-fcf1ee6c0207-metrics-tls\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689264    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689292    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-script-lib\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689311    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/de295ab7-6640-4744-826b-e780a3d7e597-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nnfbg\" (UID: \"de295ab7-6640-4744-826b-e780a3d7e597\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689344    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689344    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d81e86-e9cb-4bf1-9286-74af62e818ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689398    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689427    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689458    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-socket-dir-parent\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689492    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cni-binary-copy\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689519    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9789942-84f9-4af9-ba2d-52d3115b5882-ovnkube-script-lib\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689526    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/de295ab7-6640-4744-826b-e780a3d7e597-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nnfbg\" (UID: \"de295ab7-6640-4744-826b-e780a3d7e597\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:20:20.690663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.689528    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.691376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691359    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:20:20.691427 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691409    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 20:20:20.691462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691423    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:20:20.691462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691427    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:20:20.691671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691657    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:20:20.691764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691706    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 20:20:20.691813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691777    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 20:20:20.691857 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691840    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6mkkt\""
Oct 30 20:20:20.691932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691916    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:20:20.691932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691928    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tcbkk\""
Oct 30 20:20:20.692026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691959    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:20:20.692026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.691973    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:20:20.692108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.692030    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:20:20.692108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.692039    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-199bhgmkq2vek\""
Oct 30 20:20:20.692880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.692868    5967 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:20:20.692915 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:20.692907    5967 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.694071 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.694057    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.695213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.695198    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.695213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.695203    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.696195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.696179    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.696394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.696381    5967 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:20:20.696443 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:20.696433    5967 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.us-west-1.compute.internal"
Oct 30 20:20:20.696557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.696503    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.696730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.696708    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.697773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.697030    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.698330 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.698313    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.698412 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.698317    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.699478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.699458    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.699478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.699469    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-grpc-tls\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.699580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.699469    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.732648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.732585    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:20:20.732648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.732601    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.732648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.732645    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.732995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.732969    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:20:20.732995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.732971    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:20:20.732995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733006    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.733203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733105    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 20:20:20.733203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733140    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.733203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733179    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 20:20:20.733203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733181    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 20:20:20.733396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733358    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:20:20.733396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733364    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:20:20.733396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733383    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:20:20.733396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733388    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.733563 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733403    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.733641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733629    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 20:20:20.733641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733632    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:20:20.733722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733649    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:20:20.733722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733694    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:20:20.733827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733752    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 20:20:20.733891 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.733831    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.734398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.734377    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.734523 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.734509    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.734911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.734899    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.734958 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.734944    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.736596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736513    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.736596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736525    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f575738a-d51b-460e-9ad5-e2aeb948f4bc-metrics-client-ca\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.736596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736542    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75417679-7317-484c-bc01-0d7870ae7825-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.736596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736529    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.736596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736561    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.736596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736573    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.736992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736580    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.736992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.736694    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec59824c-8bdf-4099-9124-26a29ae8e876-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.737131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.737112    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.737291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.737275    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.737662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.737646    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.738615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.738587    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.738709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.738620    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95c43767-28f8-4bfc-960c-fcf1ee6c0207-config-volume\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.738709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.738688    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.738821 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.738707    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.738821 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.738714    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-metrics-client-ca\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.739666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.739650    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d81e86-e9cb-4bf1-9286-74af62e818ee-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.739718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.739665    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.739776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.739751    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.784807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.784403    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:20:20.784807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.784428    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:20:20.785034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.784857    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:20:20.785034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.784968    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785140    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785162    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785183    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785183    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c546q\" (UniqueName: \"kubernetes.io/projected/2f19916b-62f3-4a81-ab47-4cde132463c5-kube-api-access-c546q\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785288    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6cb\" (UniqueName: \"kubernetes.io/projected/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-kube-api-access-tr6cb\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785296    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785143    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9sm\" (UniqueName: \"kubernetes.io/projected/ec59824c-8bdf-4099-9124-26a29ae8e876-kube-api-access-6j9sm\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785471    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785532    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkff\" (UniqueName: \"kubernetes.io/projected/e97088a0-b646-48d9-818b-c932841333e4-kube-api-access-jkkff\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785683    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8z2\" (UniqueName: \"kubernetes.io/projected/75417679-7317-484c-bc01-0d7870ae7825-kube-api-access-6b8z2\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.785930    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786036    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cv52\" (UniqueName: \"kubernetes.io/projected/79d81e86-e9cb-4bf1-9286-74af62e818ee-kube-api-access-8cv52\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786059    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc44z\" (UniqueName: \"kubernetes.io/projected/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-kube-api-access-nc44z\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786078    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w25b\" (UniqueName: \"kubernetes.io/projected/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-api-access-4w25b\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786084    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786093    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vsd\" (UniqueName: \"kubernetes.io/projected/c9789942-84f9-4af9-ba2d-52d3115b5882-kube-api-access-89vsd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786039    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrfg\" (UniqueName: \"kubernetes.io/projected/d9185192-48b5-4f1d-9e4c-8344f4c64058-kube-api-access-fjrfg\") pod \"ingress-canary-x9nxm\" (UID: \"d9185192-48b5-4f1d-9e4c-8344f4c64058\") " pod="openshift-ingress-canary/ingress-canary-x9nxm"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786169    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkkh\" (UniqueName: \"kubernetes.io/projected/cb0e115f-349e-4f69-b643-336b58e3c56c-kube-api-access-gvkkh\") pod \"network-metrics-daemon-8ss9g\" (UID: \"cb0e115f-349e-4f69-b643-336b58e3c56c\") " pod="openshift-multus/network-metrics-daemon-8ss9g"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786281    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786298    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786397    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786507    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tlxf9\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786599    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786678    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786707    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxg45\" (UniqueName: \"kubernetes.io/projected/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-kube-api-access-kxg45\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786733    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxlb\" (UniqueName: \"kubernetes.io/projected/f575738a-d51b-460e-9ad5-e2aeb948f4bc-kube-api-access-qhxlb\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786777    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786797    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786850    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786872    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-svhlw\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786888    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktlv\" (UniqueName: \"kubernetes.io/projected/de0184b4-b27e-4bcf-8a5f-dacb354d2574-kube-api-access-bktlv\") pod \"network-check-target-tj2kt\" (UID: \"de0184b4-b27e-4bcf-8a5f-dacb354d2574\") " pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786918    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 20:20:20.786955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786923    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787022    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zjnsc\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786708    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrsz\" (UniqueName: \"kubernetes.io/projected/213915b5-832d-44c7-a046-12833d574630-kube-api-access-ncrsz\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787484    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkwp\" (UniqueName: \"kubernetes.io/projected/707f544c-cf0c-4c54-bd0f-6dec8791c59f-kube-api-access-wqkwp\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787613    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787763    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787785    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ldxdn\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787785    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787785    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/79d81e86-e9cb-4bf1-9286-74af62e818ee-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-f424d\" (UID: \"79d81e86-e9cb-4bf1-9286-74af62e818ee\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-f424d"
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787840    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787826    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787849    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.786850    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqzd\" (UniqueName: \"kubernetes.io/projected/81ab7c91-f29a-4204-b268-5407d7129914-kube-api-access-jtqzd\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787909    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ophv587cgj3g\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787923    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h255w\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787935    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787941    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787982    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787925    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787937    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fx2fq\""
Oct 30 20:20:20.788443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.787985    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788552    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788560    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788631    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788658    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjflc\" (UniqueName: \"kubernetes.io/projected/b4bac513-58af-4a06-a6e6-f419b8a9bc60-kube-api-access-hjflc\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788776    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788817    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a968a685-34c9-4b22-9bd6-0963e6f2dc5a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9v82b\" (UID: \"a968a685-34c9-4b22-9bd6-0963e6f2dc5a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9v82b"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.788854    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-tls\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.789142    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.789257    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdq4\" (UniqueName: \"kubernetes.io/projected/95c43767-28f8-4bfc-960c-fcf1ee6c0207-kube-api-access-6rdq4\") pod \"dns-default-b42wp\" (UID: \"95c43767-28f8-4bfc-960c-fcf1ee6c0207\") " pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:20.789366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.789332    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.789948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.789450    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec59824c-8bdf-4099-9124-26a29ae8e876-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7rhr6\" (UID: \"ec59824c-8bdf-4099-9124-26a29ae8e876\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7rhr6"
Oct 30 20:20:20.790033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.790008    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.790672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.790235    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.790672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.790480    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9jp6\""
Oct 30 20:20:20.790672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.790481    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.790672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.790602    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.790979    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-kubelet\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791020    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-hostroot\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791048    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791077    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-cnibin\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791109    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-kubelet\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791139    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-socket-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791162    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791172    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-bin\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791199    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-registration-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791230    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-sys\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791234    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-cnibin\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791266    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-kubelet-config\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791282    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-kubelet\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791303    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-var-lib-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791335    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-hostroot\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791334    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-slash\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791383    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791416    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-wtmp\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791419    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-kubelet\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791443    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-etc-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791472    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-etc-kubernetes\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791485    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-socket-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791499    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791530    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-sys\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791537    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-bin\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791560    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-dbus\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791580    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-registration-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791592    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cnibin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791624    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-systemd\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791625    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-sys\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791655    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-k8s-cni-cncf-io\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791675    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-var-lib-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791687    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791711    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75417679-7317-484c-bc01-0d7870ae7825-crio-socket\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791718    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791760    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-node-log\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791779    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791790    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-netns\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791831    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-netns\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791836    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791872    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-system-cni-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791902    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-device-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791931    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213915b5-832d-44c7-a046-12833d574630-host-slash\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791953    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-run\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791956    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792017    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792022    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-sys\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792064    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-run\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.791383    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-slash\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792083    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-dbus\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792119    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-etc-kubernetes\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792141    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-node-exporter-wtmp\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792151    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-root\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792183    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-bin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792214    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-netns\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792226    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-cnibin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792248    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4bac513-58af-4a06-a6e6-f419b8a9bc60-hosts-file\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792268    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-systemd\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792280    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/707f544c-cf0c-4c54-bd0f-6dec8791c59f-host\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792307    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-kubernetes\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792314    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f489b9e7-d156-4369-9afc-b38d675d1a31-kubelet-config\") pod \"global-pull-secret-syncer-f84bz\" (UID: \"f489b9e7-d156-4369-9afc-b38d675d1a31\") " pod="kube-system/global-pull-secret-syncer-f84bz"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792351    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-log-socket\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792367    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-k8s-cni-cncf-io\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792382    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792408    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81ab7c91-f29a-4204-b268-5407d7129914-root\") pod \"node-exporter-l9bdp\" (UID: \"81ab7c91-f29a-4204-b268-5407d7129914\") " pod="openshift-monitoring/node-exporter-l9bdp"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792414    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-lib-modules\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792450    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-bin\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792449    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-socket-dir-parent\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792487    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-var-lib-kubelet\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792493    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-socket-dir-parent\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792522    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysconfig\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792549    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-conf-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792563    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792589    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-multus\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792610    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75417679-7317-484c-bc01-0d7870ae7825-crio-socket\") pod \"insights-runtime-extractor-9zjw5\" (UID: \"75417679-7317-484c-bc01-0d7870ae7825\") " pod="openshift-insights/insights-runtime-extractor-9zjw5"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792618    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-systemd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792652    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-systemd-units\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792657    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-systemd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792184    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-etc-openvswitch\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792682    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-ovn\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792708    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-run-netns\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792711    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-netd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792780    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-cni-netd\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792802    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792842    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-modprobe-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792847    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-node-log\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792874    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-conf\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792895    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f19916b-62f3-4a81-ab47-4cde132463c5-system-cni-dir\") pod \"multus-additional-cni-plugins-rk542\" (UID: \"2f19916b-62f3-4a81-ab47-4cde132463c5\") " pod="openshift-multus/multus-additional-cni-plugins-rk542"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792913    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792945    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-sys-fs\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792948    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-device-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792970    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-system-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.792990    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213915b5-832d-44c7-a046-12833d574630-host-slash\") pod \"iptables-alerter-dgvmc\" (UID: \"213915b5-832d-44c7-a046-12833d574630\") " pod="openshift-network-operator/iptables-alerter-dgvmc"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793002    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-multus-certs\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793033    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-host\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793039    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-kubernetes\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793125    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-conf-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793171    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-var-lib-cni-multus\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793197    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-modprobe-d\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793223    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793278    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-lib-modules\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793337    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4bac513-58af-4a06-a6e6-f419b8a9bc60-hosts-file\") pod \"node-resolver-blv5n\" (UID: \"b4bac513-58af-4a06-a6e6-f419b8a9bc60\") " pod="openshift-dns/node-resolver-blv5n"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793341    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793357    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysconfig\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.793303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793359    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-run-ovn\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793438    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-var-lib-kubelet\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793487    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-systemd-units\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793501    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-host\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793526    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/707f544c-cf0c-4c54-bd0f-6dec8791c59f-host\") pod \"node-ca-qtnkw\" (UID: \"707f544c-cf0c-4c54-bd0f-6dec8791c59f\") " pod="openshift-image-registry/node-ca-qtnkw"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793557    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-system-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793564    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9789942-84f9-4af9-ba2d-52d3115b5882-log-socket\") pod \"ovnkube-node-7pz42\" (UID: \"c9789942-84f9-4af9-ba2d-52d3115b5882\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793598    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-host-run-multus-certs\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793610    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97-multus-cni-dir\") pod \"multus-cfstj\" (UID: \"a2fec4f0-9aa9-4e1f-be2e-66c8c4968e97\") " pod="openshift-multus/multus-cfstj"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793660    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e97088a0-b646-48d9-818b-c932841333e4-sys-fs\") pod \"aws-ebs-csi-driver-node-bkf4p\" (UID: \"e97088a0-b646-48d9-818b-c932841333e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bkf4p"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.793832    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f681324-576a-4ae5-95fd-2a7e8f8ce67a-etc-sysctl-conf\") pod \"tuned-jw4mv\" (UID: \"1f681324-576a-4ae5-95fd-2a7e8f8ce67a\") " pod="openshift-cluster-node-tuning-operator/tuned-jw4mv"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.794280    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.794323    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff0b92cc-28ad-435b-bbcb-161bb00e2c42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-lwh2w\" (UID: \"ff0b92cc-28ad-435b-bbcb-161bb00e2c42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.794376    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") pod \"console-5466579cfb-k26jn\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") " pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.794422    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.795154    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.795303    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.795369    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.796110    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f575738a-d51b-460e-9ad5-e2aeb948f4bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c64476cfd-sv4hq\" (UID: \"f575738a-d51b-460e-9ad5-e2aeb948f4bc\") " pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.796131    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.796468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.796193    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") pod \"alertmanager-main-0\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:20:20.797318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.797301    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") pod \"console-76f877b658-k6n9x\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") " pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:20.797409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.797397    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") pod \"console-c79948c48-z5442\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") " pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:20.894998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.894975    5967 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:20:20.997046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:20.996960    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f877b658-k6n9x"]
Oct 30 20:20:21.046474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:21.046431    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:21.052205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:21.052185    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c64476cfd-sv4hq"
Oct 30 20:20:21.599725 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:21.599692    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b42wp"
Oct 30 20:20:23.672402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:23.672366    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:20:24.236010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:24.235975    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:24.638059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:24.637958    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"}
Oct 30 20:20:24.638059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:24.638022    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"}
Oct 30 20:20:24.735545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:24.735509    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:24.740474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:24.740445    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-lwh2w"
Oct 30 20:20:25.483995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:25.483963    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nnfbg"
Oct 30 20:20:25.644712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:25.644674    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"}
Oct 30 20:20:25.644908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:25.644769    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"}
Oct 30 20:20:25.644908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:25.644820    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"}
Oct 30 20:20:25.644908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:25.644849    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerStarted","Data":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"}
Oct 30 20:20:26.902648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:26.902617    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:27.290790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:27.290734    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:28.759782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:28.759686    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:28.760318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:28.759964    5967 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:20:29.902402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:29.902368    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:30.606050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:30.606017    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:31.980267 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:31.980232    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:20:34.240828 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:34.240783    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:34.244340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:34.244318    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:20:35.814311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:35.814285    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pxcqk"
Oct 30 20:20:38.556755 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:38.556712    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pz42"
Oct 30 20:20:41.697304 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:41.697268    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:41.716571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:41.716543    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:41.731688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:41.731667    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:20:45.609483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.609417    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c79948c48-z5442" podUID="020e1e00-032c-4533-ba58-97b7f4971d37" containerName="console" containerID="cri-o://96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658" gracePeriod=15
Oct 30 20:20:45.860340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.860281    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:45.905235 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905204    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905259    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905295    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905351    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905388    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905430    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905457    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905524    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") pod \"020e1e00-032c-4533-ba58-97b7f4971d37\" (UID: \"020e1e00-032c-4533-ba58-97b7f4971d37\") "
Oct 30 20:20:45.905687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905524    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:45.905871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905769    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:45.905871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905825    5967 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/020e1e00-032c-4533-ba58-97b7f4971d37-tmp\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:45.905986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905928    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:45.905986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.905949    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:45.906189 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.906166    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:45.907812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.907787    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "kube-api-access-2vjj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:45.907812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.907799    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:45.907959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:45.907898    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert" pod "020e1e00-032c-4533-ba58-97b7f4971d37" (UID: "020e1e00-032c-4533-ba58-97b7f4971d37"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:46.006844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006813    5967 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-console-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.006844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006840    5967 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-oauth-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.006844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006850    5967 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-trusted-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.007065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006860    5967 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-oauth-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.007065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006868    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjj7\" (UniqueName: \"kubernetes.io/projected/020e1e00-032c-4533-ba58-97b7f4971d37-kube-api-access-2vjj7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.007065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006878    5967 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020e1e00-032c-4533-ba58-97b7f4971d37-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.007065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.006886    5967 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020e1e00-032c-4533-ba58-97b7f4971d37-console-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.640037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.639969    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76f877b658-k6n9x" podUID="51ef5c28-b16d-425a-b173-0ec514a3c3de" containerName="console" containerID="cri-o://4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73" gracePeriod=15
Oct 30 20:20:46.725562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.725517    5967 generic.go:356] "Generic (PLEG): container finished" podID="020e1e00-032c-4533-ba58-97b7f4971d37" containerID="96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658" exitCode=0
Oct 30 20:20:46.725757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.725567    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c79948c48-z5442"
Oct 30 20:20:46.725757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.725603    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c79948c48-z5442" event={"ID":"020e1e00-032c-4533-ba58-97b7f4971d37","Type":"ContainerDied","Data":"96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658"}
Oct 30 20:20:46.725757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.725649    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c79948c48-z5442" event={"ID":"020e1e00-032c-4533-ba58-97b7f4971d37","Type":"ContainerDied","Data":"da8fecc3894f9db23c9a705d1fe7ad7c431fc36dc4bc04fb5bb3f2558ef9637f"}
Oct 30 20:20:46.725757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.725668    5967 scope.go:122] "RemoveContainer" containerID="96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658"
Oct 30 20:20:46.750858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.750839    5967 scope.go:122] "RemoveContainer" containerID="96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658"
Oct 30 20:20:46.751241 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:46.751220    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658\": container with ID starting with 96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658 not found: ID does not exist" containerID="96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658"
Oct 30 20:20:46.751289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.751250    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658"} err="failed to get container status \"96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658\": rpc error: code = NotFound desc = could not find container \"96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658\": container with ID starting with 96eda1ff064a08d66a027eafa4752b6d5eee6d5b65bf4c893696f755da2c6658 not found: ID does not exist"
Oct 30 20:20:46.887364 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.887340    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c79948c48-z5442"]
Oct 30 20:20:46.899827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.899762    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:46.936808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:46.936771    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c79948c48-z5442"]
Oct 30 20:20:47.015199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015165    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015220    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015259    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015286    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015373    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015415    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015458    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") pod \"51ef5c28-b16d-425a-b173-0ec514a3c3de\" (UID: \"51ef5c28-b16d-425a-b173-0ec514a3c3de\") "
Oct 30 20:20:47.015787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.015660    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:47.016064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.016008    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:47.016765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.016700    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:47.017699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.017673    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:47.018322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.018296    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "kube-api-access-qgnn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:47.018484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.018456    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:47.018630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.018607    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert" pod "51ef5c28-b16d-425a-b173-0ec514a3c3de" (UID: "51ef5c28-b16d-425a-b173-0ec514a3c3de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:47.116810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116770    5967 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-oauth-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.116810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116796    5967 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-oauth-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.116810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116814    5967 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.117118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116823    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnn2\" (UniqueName: \"kubernetes.io/projected/51ef5c28-b16d-425a-b173-0ec514a3c3de-kube-api-access-qgnn2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.117118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116854    5967 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51ef5c28-b16d-425a-b173-0ec514a3c3de-console-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.117118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116866    5967 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef5c28-b16d-425a-b173-0ec514a3c3de-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.117118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.116876    5967 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51ef5c28-b16d-425a-b173-0ec514a3c3de-tmp\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:47.540167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.540137    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020e1e00-032c-4533-ba58-97b7f4971d37" path="/var/lib/kubelet/pods/020e1e00-032c-4533-ba58-97b7f4971d37/volumes"
Oct 30 20:20:47.730887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.730856    5967 generic.go:356] "Generic (PLEG): container finished" podID="51ef5c28-b16d-425a-b173-0ec514a3c3de" containerID="4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73" exitCode=0
Oct 30 20:20:47.731366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.730903    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f877b658-k6n9x"
Oct 30 20:20:47.731366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.730942    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f877b658-k6n9x" event={"ID":"51ef5c28-b16d-425a-b173-0ec514a3c3de","Type":"ContainerDied","Data":"4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73"}
Oct 30 20:20:47.731366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.731000    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f877b658-k6n9x" event={"ID":"51ef5c28-b16d-425a-b173-0ec514a3c3de","Type":"ContainerDied","Data":"71fc723cf58149f2cff1e5960ddb8bb0018584ec1a4889b63c458bc722462fd2"}
Oct 30 20:20:47.731366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.731028    5967 scope.go:122] "RemoveContainer" containerID="4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73"
Oct 30 20:20:47.740224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.740200    5967 scope.go:122] "RemoveContainer" containerID="4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73"
Oct 30 20:20:47.740478 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:20:47.740462    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73\": container with ID starting with 4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73 not found: ID does not exist" containerID="4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73"
Oct 30 20:20:47.740536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.740484    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73"} err="failed to get container status \"4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73\": rpc error: code = NotFound desc = could not find container \"4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73\": container with ID starting with 4158ca52e201cfc3a8828d1e7b0686c98fcc44516e6825b2ee94b61ec4bc1d73 not found: ID does not exist"
Oct 30 20:20:47.892720 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.892642    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f877b658-k6n9x"]
Oct 30 20:20:47.942206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:47.942169    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76f877b658-k6n9x"]
Oct 30 20:20:49.540295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:49.540264    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ef5c28-b16d-425a-b173-0ec514a3c3de" path="/var/lib/kubelet/pods/51ef5c28-b16d-425a-b173-0ec514a3c3de/volumes"
Oct 30 20:20:53.675308 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:20:53.675271    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tj2kt"
Oct 30 20:21:00.772175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.772146    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:21:00.772700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.772446    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-metric" containerID="cri-o://070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72" gracePeriod=120
Oct 30 20:21:00.772700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.772513    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-web" containerID="cri-o://840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b" gracePeriod=120
Oct 30 20:21:00.772700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.772598    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="prom-label-proxy" containerID="cri-o://674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709" gracePeriod=120
Oct 30 20:21:00.772700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.772421    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="alertmanager" containerID="cri-o://4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2" gracePeriod=120
Oct 30 20:21:00.772700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.772672    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="config-reloader" containerID="cri-o://bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625" gracePeriod=120
Oct 30 20:21:00.773061 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:00.773031    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy" containerID="cri-o://e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99" gracePeriod=120
Oct 30 20:21:01.792210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792178    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709" exitCode=0
Oct 30 20:21:01.792210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792203    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99" exitCode=0
Oct 30 20:21:01.792210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792209    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625" exitCode=0
Oct 30 20:21:01.792210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792215    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2" exitCode=0
Oct 30 20:21:01.792681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792249    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"}
Oct 30 20:21:01.792681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792311    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"}
Oct 30 20:21:01.792681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792339    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"}
Oct 30 20:21:01.792681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:01.792362    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"}
Oct 30 20:21:01.981129 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:01.981085    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:21:02.023207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.023182    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:02.173251 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173175    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173251 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173217    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173258    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173288    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173324    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173370    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173415    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173459    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173492    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173525    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173555    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173585    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173661    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") pod \"1df472f3-8947-48ba-a2d7-7e4ce4786505\" (UID: \"1df472f3-8947-48ba-a2d7-7e4ce4786505\") "
Oct 30 20:21:02.173805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173696    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:02.174166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.173957    5967 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-main-db\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.175090 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.175014    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:02.175541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.175514    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:02.176470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.176441    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.177384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.177355    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.177643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.177615    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:02.177811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.177634    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.178000 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.177708    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "kube-api-access-7z2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:02.178000 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.177812    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.178158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.178104    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:02.178667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.178644    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.182100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.182070    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.188610 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.188586    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config" pod "1df472f3-8947-48ba-a2d7-7e4ce4786505" (UID: "1df472f3-8947-48ba-a2d7-7e4ce4786505"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:02.274354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274319    5967 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-cluster-tls-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274347    5967 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-main-tls\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274375    5967 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-web-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274384    5967 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-out\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274392    5967 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-metrics-client-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274401    5967 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df472f3-8947-48ba-a2d7-7e4ce4786505-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274410    5967 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274419    5967 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274428    5967 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-tls-assets\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274436    5967 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274444    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/1df472f3-8947-48ba-a2d7-7e4ce4786505-kube-api-access-7z2q9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.274571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.274455    5967 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1df472f3-8947-48ba-a2d7-7e4ce4786505-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:02.799298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799265    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72" exitCode=0
Oct 30 20:21:02.799298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799291    5967 generic.go:356] "Generic (PLEG): container finished" podID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerID="840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b" exitCode=0
Oct 30 20:21:02.799792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799373    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"}
Oct 30 20:21:02.799792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799414    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:02.799792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799432    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"}
Oct 30 20:21:02.799792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799464    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1df472f3-8947-48ba-a2d7-7e4ce4786505","Type":"ContainerDied","Data":"86a304bb9a84088b3c8574cfd4a7242c4a98493e8ab512a8f19c99cbb583e122"}
Oct 30 20:21:02.799792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.799492    5967 scope.go:122] "RemoveContainer" containerID="674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"
Oct 30 20:21:02.807627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.807610    5967 scope.go:122] "RemoveContainer" containerID="070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"
Oct 30 20:21:02.814894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.814875    5967 scope.go:122] "RemoveContainer" containerID="e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"
Oct 30 20:21:02.821589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.821572    5967 scope.go:122] "RemoveContainer" containerID="840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"
Oct 30 20:21:02.828209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.828192    5967 scope.go:122] "RemoveContainer" containerID="bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"
Oct 30 20:21:02.835205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.835185    5967 scope.go:122] "RemoveContainer" containerID="4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"
Oct 30 20:21:02.842612 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.842592    5967 scope.go:122] "RemoveContainer" containerID="cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"
Oct 30 20:21:02.849309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.849282    5967 scope.go:122] "RemoveContainer" containerID="674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"
Oct 30 20:21:02.849546 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.849529    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709\": container with ID starting with 674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709 not found: ID does not exist" containerID="674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"
Oct 30 20:21:02.849589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.849559    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"} err="failed to get container status \"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709\": rpc error: code = NotFound desc = could not find container \"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709\": container with ID starting with 674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709 not found: ID does not exist"
Oct 30 20:21:02.849589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.849578    5967 scope.go:122] "RemoveContainer" containerID="070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"
Oct 30 20:21:02.849928 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.849909    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72\": container with ID starting with 070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72 not found: ID does not exist" containerID="070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"
Oct 30 20:21:02.850009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.849937    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"} err="failed to get container status \"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72\": rpc error: code = NotFound desc = could not find container \"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72\": container with ID starting with 070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72 not found: ID does not exist"
Oct 30 20:21:02.850009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.849962    5967 scope.go:122] "RemoveContainer" containerID="e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"
Oct 30 20:21:02.850182 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.850166    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99\": container with ID starting with e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99 not found: ID does not exist" containerID="e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"
Oct 30 20:21:02.850240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850189    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"} err="failed to get container status \"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99\": rpc error: code = NotFound desc = could not find container \"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99\": container with ID starting with e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99 not found: ID does not exist"
Oct 30 20:21:02.850240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850208    5967 scope.go:122] "RemoveContainer" containerID="840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"
Oct 30 20:21:02.850430 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.850410    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b\": container with ID starting with 840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b not found: ID does not exist" containerID="840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"
Oct 30 20:21:02.850469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850438    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"} err="failed to get container status \"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b\": rpc error: code = NotFound desc = could not find container \"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b\": container with ID starting with 840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b not found: ID does not exist"
Oct 30 20:21:02.850469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850451    5967 scope.go:122] "RemoveContainer" containerID="bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"
Oct 30 20:21:02.850681 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.850661    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625\": container with ID starting with bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625 not found: ID does not exist" containerID="bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"
Oct 30 20:21:02.850728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850686    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"} err="failed to get container status \"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625\": rpc error: code = NotFound desc = could not find container \"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625\": container with ID starting with bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625 not found: ID does not exist"
Oct 30 20:21:02.850728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850700    5967 scope.go:122] "RemoveContainer" containerID="4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"
Oct 30 20:21:02.850949 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.850933    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2\": container with ID starting with 4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2 not found: ID does not exist" containerID="4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"
Oct 30 20:21:02.851011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850956    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"} err="failed to get container status \"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2\": rpc error: code = NotFound desc = could not find container \"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2\": container with ID starting with 4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2 not found: ID does not exist"
Oct 30 20:21:02.851011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.850974    5967 scope.go:122] "RemoveContainer" containerID="cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"
Oct 30 20:21:02.851190 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:02.851175    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6\": container with ID starting with cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6 not found: ID does not exist" containerID="cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"
Oct 30 20:21:02.851226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851194    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"} err="failed to get container status \"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6\": rpc error: code = NotFound desc = could not find container \"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6\": container with ID starting with cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6 not found: ID does not exist"
Oct 30 20:21:02.851226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851208    5967 scope.go:122] "RemoveContainer" containerID="674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"
Oct 30 20:21:02.851401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851387    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709"} err="failed to get container status \"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709\": rpc error: code = NotFound desc = could not find container \"674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709\": container with ID starting with 674bbbf2e76d514b9cc8fd7fea7dbced350932f9c4136ee428488e2be6dea709 not found: ID does not exist"
Oct 30 20:21:02.851444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851401    5967 scope.go:122] "RemoveContainer" containerID="070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"
Oct 30 20:21:02.851600 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851583    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72"} err="failed to get container status \"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72\": rpc error: code = NotFound desc = could not find container \"070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72\": container with ID starting with 070953bc31ac04ffde13a108680dcbb340b0b0129492e50f5734853ec5ae3a72 not found: ID does not exist"
Oct 30 20:21:02.851665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851601    5967 scope.go:122] "RemoveContainer" containerID="e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"
Oct 30 20:21:02.851858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851843    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99"} err="failed to get container status \"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99\": rpc error: code = NotFound desc = could not find container \"e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99\": container with ID starting with e07ea69dfcf20db0cb14f3b2729d547de79353ab9a92aec071ae3e33aa389d99 not found: ID does not exist"
Oct 30 20:21:02.851917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.851859    5967 scope.go:122] "RemoveContainer" containerID="840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"
Oct 30 20:21:02.852098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852078    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b"} err="failed to get container status \"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b\": rpc error: code = NotFound desc = could not find container \"840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b\": container with ID starting with 840464ecfa850ab9f9b44e501c9a891420a9de2a93729f5d2d15f7bd570c675b not found: ID does not exist"
Oct 30 20:21:02.852145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852098    5967 scope.go:122] "RemoveContainer" containerID="bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"
Oct 30 20:21:02.852294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852279    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625"} err="failed to get container status \"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625\": rpc error: code = NotFound desc = could not find container \"bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625\": container with ID starting with bad2f239bbb40a36cdccf8235cd24457a6cce853819d7e8793b2046adf48e625 not found: ID does not exist"
Oct 30 20:21:02.852294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852294    5967 scope.go:122] "RemoveContainer" containerID="4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"
Oct 30 20:21:02.852498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852479    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2"} err="failed to get container status \"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2\": rpc error: code = NotFound desc = could not find container \"4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2\": container with ID starting with 4bd700937ff025bdd7b4b77041f6f55254d6ded3d711e2878d8350a85122dde2 not found: ID does not exist"
Oct 30 20:21:02.852542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852499    5967 scope.go:122] "RemoveContainer" containerID="cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"
Oct 30 20:21:02.852729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:02.852709    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6"} err="failed to get container status \"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6\": rpc error: code = NotFound desc = could not find container \"cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6\": container with ID starting with cac3701cccb45027763b5e7f8ce001ea0a9fa97b7f5b0bcae2ce5f64f08fcda6 not found: ID does not exist"
Oct 30 20:21:03.013927 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.013893    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:21:03.063910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.063772    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:21:03.064418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064393    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:21:03.064857 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064839    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="config-reloader"
Oct 30 20:21:03.064935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064879    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="config-reloader"
Oct 30 20:21:03.064935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064901    5967 cpu_manager.go:414] "containerMap: removing container" podUID="020e1e00-032c-4533-ba58-97b7f4971d37" containerName="console"
Oct 30 20:21:03.064935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064909    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020e1e00-032c-4533-ba58-97b7f4971d37" containerName="console"
Oct 30 20:21:03.064935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064918    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="prom-label-proxy"
Oct 30 20:21:03.064935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064925    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="prom-label-proxy"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064944    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-web"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064952    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-web"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064962    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064967    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064982    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="alertmanager"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064986    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="alertmanager"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.064996    5967 cpu_manager.go:414] "containerMap: removing container" podUID="51ef5c28-b16d-425a-b173-0ec514a3c3de" containerName="console"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065000    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ef5c28-b16d-425a-b173-0ec514a3c3de" containerName="console"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065007    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-metric"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065012    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-metric"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065018    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="init-config-reloader"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065023    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="init-config-reloader"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065079    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="alertmanager"
Oct 30 20:21:03.065082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065088    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ef5c28-b16d-425a-b173-0ec514a3c3de" containerName="console"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065094    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-metric"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065100    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="init-config-reloader"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065109    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="config-reloader"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065117    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="020e1e00-032c-4533-ba58-97b7f4971d37" containerName="console"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065124    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="prom-label-proxy"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065132    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy-web"
Oct 30 20:21:03.065455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.065139    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" containerName="kube-rbac-proxy"
Oct 30 20:21:03.068932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.068916    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.121379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121348    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:21:03.121661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121637    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:21:03.121661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121664    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:21:03.121924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121704    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:21:03.121924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121717    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:21:03.121924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121710    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ldxdn\""
Oct 30 20:21:03.121924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121792    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:21:03.121924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121897    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:21:03.121924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.121909    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:21:03.171045 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.171011    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:21:03.181221 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181196    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-web-config\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181227    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181251    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0811cb0-c2cc-4560-a61e-1f2077870d7f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181270    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0811cb0-c2cc-4560-a61e-1f2077870d7f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181369    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0811cb0-c2cc-4560-a61e-1f2077870d7f-config-out\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181393    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181469    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0811cb0-c2cc-4560-a61e-1f2077870d7f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181493    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shr82\" (UniqueName: \"kubernetes.io/projected/b0811cb0-c2cc-4560-a61e-1f2077870d7f-kube-api-access-shr82\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181613 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181559    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0811cb0-c2cc-4560-a61e-1f2077870d7f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181613 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181592    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181616    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181670    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.181843 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.181711    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.282884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.282844    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-web-config\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.282890    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283021    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0811cb0-c2cc-4560-a61e-1f2077870d7f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283064    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0811cb0-c2cc-4560-a61e-1f2077870d7f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283084    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0811cb0-c2cc-4560-a61e-1f2077870d7f-config-out\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283108    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283136    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0811cb0-c2cc-4560-a61e-1f2077870d7f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283158    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shr82\" (UniqueName: \"kubernetes.io/projected/b0811cb0-c2cc-4560-a61e-1f2077870d7f-kube-api-access-shr82\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283189    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0811cb0-c2cc-4560-a61e-1f2077870d7f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283218    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283253    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283279    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283307    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.283582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283348    5967 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:21:03.283785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.283615    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b0811cb0-c2cc-4560-a61e-1f2077870d7f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.284602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.284430    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0811cb0-c2cc-4560-a61e-1f2077870d7f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.284602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.284576    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0811cb0-c2cc-4560-a61e-1f2077870d7f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.286505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.286477    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0811cb0-c2cc-4560-a61e-1f2077870d7f-config-out\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.286608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.286480    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.286608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.286566    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.286608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.286568    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.286790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.286649    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.287103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.287078    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.287235 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.287215    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-web-config\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.287291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.287258    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0811cb0-c2cc-4560-a61e-1f2077870d7f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.288571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.288552    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0811cb0-c2cc-4560-a61e-1f2077870d7f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.298244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.298214    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:21:03.339704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.339638    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shr82\" (UniqueName: \"kubernetes.io/projected/b0811cb0-c2cc-4560-a61e-1f2077870d7f-kube-api-access-shr82\") pod \"alertmanager-main-0\" (UID: \"b0811cb0-c2cc-4560-a61e-1f2077870d7f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.385349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.385317    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:21:03.540221 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.540186    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df472f3-8947-48ba-a2d7-7e4ce4786505" path="/var/lib/kubelet/pods/1df472f3-8947-48ba-a2d7-7e4ce4786505/volumes"
Oct 30 20:21:03.811954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:03.811927    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:21:03.814782 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:21:03.814750    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0811cb0_c2cc_4560_a61e_1f2077870d7f.slice/crio-f47e9a8b6b802a641b97e3d378cdceb94e572a7c64cba3f4998e95bb5fc93a40 WatchSource:0}: Error finding container f47e9a8b6b802a641b97e3d378cdceb94e572a7c64cba3f4998e95bb5fc93a40: Status 404 returned error can't find the container with id f47e9a8b6b802a641b97e3d378cdceb94e572a7c64cba3f4998e95bb5fc93a40
Oct 30 20:21:04.809516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:04.809481    5967 generic.go:356] "Generic (PLEG): container finished" podID="b0811cb0-c2cc-4560-a61e-1f2077870d7f" containerID="4ea5c9db83634791772d3ac7c4b3a0ac5b221ca228237df495664466946308e8" exitCode=0
Oct 30 20:21:04.809676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:04.809557    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerDied","Data":"4ea5c9db83634791772d3ac7c4b3a0ac5b221ca228237df495664466946308e8"}
Oct 30 20:21:04.809676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:04.809625    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"f47e9a8b6b802a641b97e3d378cdceb94e572a7c64cba3f4998e95bb5fc93a40"}
Oct 30 20:21:05.816632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.816602    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"aef44f14836ff8cec3ba918c69e8bcdb592cb545fbf293d5ebfa7279e5675cf1"}
Oct 30 20:21:05.817106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.816659    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"66a3d0a665655b287e291c633bd73c1c99516fa4bce8bc9fe187c7270b3575f6"}
Oct 30 20:21:05.817106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.816686    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"834f8cff60b2d198be5b46fb411553e288cf7ae47c32f834be46e25a33c6763c"}
Oct 30 20:21:05.817106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.816713    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"67e56ec789b144366beffc056d6c4496c252996eb341d51a814fdbde83d2aea4"}
Oct 30 20:21:05.817106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.816735    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"2509b756c589ed2724704976e34d1f0ecf2b5caa85a97f88121583bfc0bb747b"}
Oct 30 20:21:05.817106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.816785    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b0811cb0-c2cc-4560-a61e-1f2077870d7f","Type":"ContainerStarted","Data":"1cb739905c6d587d268f2478f49c0a81bbf1156edf5ae395d8a61635e8e7093d"}
Oct 30 20:21:05.946655 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:05.946592    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.934404278 podStartE2EDuration="2.946580197s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.175919ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:05.94533279 +0000 UTC m=+46.546134751" watchObservedRunningTime="2026-10-30 20:21:05.946580197 +0000 UTC m=+46.547382153"
Oct 30 20:21:06.438518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438478    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:21:06.439131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438867    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="prometheus" containerID="cri-o://e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622" gracePeriod=600
Oct 30 20:21:06.439131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438960    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="config-reloader" containerID="cri-o://2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4" gracePeriod=600
Oct 30 20:21:06.439131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438885    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="thanos-sidecar" containerID="cri-o://d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652" gracePeriod=600
Oct 30 20:21:06.439131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438943    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a" gracePeriod=600
Oct 30 20:21:06.439131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438937    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-web" containerID="cri-o://6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291" gracePeriod=600
Oct 30 20:21:06.439428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.438893    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy" containerID="cri-o://4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4" gracePeriod=600
Oct 30 20:21:06.680031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.680005    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:06.710521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710446    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710478    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710499    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710529    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710561    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710589    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710618    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710649    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710676    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710713    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710771    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.710811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710805    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710835    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710866    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710897    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710958    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.710973    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.711001    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.711028    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.711057    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.711113    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") pod \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\" (UID: \"8fa8d11e-1a70-4b91-8270-dd8c93d8511c\") "
Oct 30 20:21:06.711414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.711358    5967 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-metrics-client-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.711913 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.711729    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.712385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.712342    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.712972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.712583    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.712972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.712762    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.713831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.713644    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.714117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.714079    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:06.714229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.714197    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.714636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.714602    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:06.715422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.715394    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "kube-api-access-l9mt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:06.715531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.715468    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.715531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.715519    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.716429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.716393    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:06.716528 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.716481    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.716663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.716637    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:06.717052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.717028    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.717159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.717141    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.717547 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.717526    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.717617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.717598    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.727017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.726996    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config" pod "8fa8d11e-1a70-4b91-8270-dd8c93d8511c" (UID: "8fa8d11e-1a70-4b91-8270-dd8c93d8511c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:06.812288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812250    5967 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-kube-rbac-proxy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812290    5967 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-grpc-tls\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812306    5967 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-web-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812320    5967 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812334    5967 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-tls-assets\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812347    5967 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-metrics-client-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812362    5967 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812377    5967 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812392    5967 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812404    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mt8\" (UniqueName: \"kubernetes.io/projected/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-kube-api-access-l9mt8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812420    5967 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812433    5967 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-db\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812445    5967 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812458    5967 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812471    5967 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-config-out\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812484    5967 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812498    5967 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812511    5967 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.812531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.812525    5967 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8fa8d11e-1a70-4b91-8270-dd8c93d8511c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:06.826042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.825997    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a" exitCode=0
Oct 30 20:21:06.826042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826039    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4" exitCode=0
Oct 30 20:21:06.826042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826049    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291" exitCode=0
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826057    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652" exitCode=0
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826065    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4" exitCode=0
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826074    5967 generic.go:356] "Generic (PLEG): container finished" podID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622" exitCode=0
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826078    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"}
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826118    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826159    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"}
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826188    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"}
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826219    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"}
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826235    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826249    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"}
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826273    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"}
Oct 30 20:21:06.826535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.826302    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8fa8d11e-1a70-4b91-8270-dd8c93d8511c","Type":"ContainerDied","Data":"97ba26ab7e909489d4985819f1756bef2f6b1c00a8164ff068dddc925fd1a7e1"}
Oct 30 20:21:06.840777 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.840728    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.848163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.848144    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.855326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.855298    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.862881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.862850    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.869610 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.869592    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.877145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.877115    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.897611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.897589    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.897972 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.897939    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.898089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.897986    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"} err="failed to get container status \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist"
Oct 30 20:21:06.898089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.898027    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.898337 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.898316    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.898422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.898358    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"} err="failed to get container status \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist"
Oct 30 20:21:06.898422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.898383    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.898716 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.898696    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.898798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.898723    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"} err="failed to get container status \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist"
Oct 30 20:21:06.898798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.898762    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.899003 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.898986    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.899116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899010    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"} err="failed to get container status \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist"
Oct 30 20:21:06.899116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899027    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.899243 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.899228    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.899284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899249    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"} err="failed to get container status \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist"
Oct 30 20:21:06.899284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899262    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.899532 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.899514    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.899568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899538    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"} err="failed to get container status \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist"
Oct 30 20:21:06.899568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899552    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.899783 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:06.899767    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.899835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899786    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"} err="failed to get container status \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist"
Oct 30 20:21:06.899835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.899799    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.900078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900060    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"} err="failed to get container status \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist"
Oct 30 20:21:06.900128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900079    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.900289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900274    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"} err="failed to get container status \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist"
Oct 30 20:21:06.900289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900288    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.900486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900470    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"} err="failed to get container status \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist"
Oct 30 20:21:06.900486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900483    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.900690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900668    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"} err="failed to get container status \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist"
Oct 30 20:21:06.900753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900691    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.900973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900954    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"} err="failed to get container status \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist"
Oct 30 20:21:06.901028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.900975    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.901206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901191    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"} err="failed to get container status \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist"
Oct 30 20:21:06.901254 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901207    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.901418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901401    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"} err="failed to get container status \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist"
Oct 30 20:21:06.901454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901419    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.901659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901641    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"} err="failed to get container status \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist"
Oct 30 20:21:06.901659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901657    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.901916 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901900    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"} err="failed to get container status \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist"
Oct 30 20:21:06.901962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.901917    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.902163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902146    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"} err="failed to get container status \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist"
Oct 30 20:21:06.902214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902163    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.902407 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902389    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"} err="failed to get container status \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist"
Oct 30 20:21:06.902456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902408    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.902644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902621    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"} err="failed to get container status \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist"
Oct 30 20:21:06.902694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902644    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.902897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902877    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"} err="failed to get container status \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist"
Oct 30 20:21:06.902973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.902897    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.903136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903120    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"} err="failed to get container status \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist"
Oct 30 20:21:06.903209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903138    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.903370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903352    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"} err="failed to get container status \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist"
Oct 30 20:21:06.903438 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903370    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.903566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903550    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"} err="failed to get container status \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist"
Oct 30 20:21:06.903628 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903567    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.903878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903858    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"} err="failed to get container status \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist"
Oct 30 20:21:06.903933 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.903879    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.904112 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904092    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"} err="failed to get container status \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist"
Oct 30 20:21:06.904146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904113    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.904339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904320    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"} err="failed to get container status \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist"
Oct 30 20:21:06.904339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904339    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.904557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904524    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"} err="failed to get container status \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist"
Oct 30 20:21:06.904595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904557    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.904792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904777    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"} err="failed to get container status \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist"
Oct 30 20:21:06.904792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.904792    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.905057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905039    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"} err="failed to get container status \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist"
Oct 30 20:21:06.905057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905057    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.905264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905247    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"} err="failed to get container status \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist"
Oct 30 20:21:06.905323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905263    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.905478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905461    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"} err="failed to get container status \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist"
Oct 30 20:21:06.905478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905477    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.905676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905660    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"} err="failed to get container status \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist"
Oct 30 20:21:06.905676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905676    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.905912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905894    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"} err="failed to get container status \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist"
Oct 30 20:21:06.905955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.905912    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.906104 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906090    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"} err="failed to get container status \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist"
Oct 30 20:21:06.906140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906104    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.906301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906283    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"} err="failed to get container status \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist"
Oct 30 20:21:06.906301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906300    5967 scope.go:122] "RemoveContainer" containerID="af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"
Oct 30 20:21:06.906493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906477    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a"} err="failed to get container status \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": rpc error: code = NotFound desc = could not find container \"af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a\": container with ID starting with af18bf5235d103880d06ffa79780c77c99b6d5461f96a3a6ccf56ab05ea8227a not found: ID does not exist"
Oct 30 20:21:06.906493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906491    5967 scope.go:122] "RemoveContainer" containerID="4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"
Oct 30 20:21:06.906682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906667    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4"} err="failed to get container status \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": rpc error: code = NotFound desc = could not find container \"4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4\": container with ID starting with 4e0f9ce4e3d65a30beb5691c3f1e1ac87e424d07af19a6dade2169c280f8a9c4 not found: ID does not exist"
Oct 30 20:21:06.906682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906680    5967 scope.go:122] "RemoveContainer" containerID="6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"
Oct 30 20:21:06.906935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906916    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291"} err="failed to get container status \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": rpc error: code = NotFound desc = could not find container \"6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291\": container with ID starting with 6b9aa667940f136a9bc5c34319a6bbe34c0c0e4ae0c2b602aa6eebf772984291 not found: ID does not exist"
Oct 30 20:21:06.906979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.906935    5967 scope.go:122] "RemoveContainer" containerID="d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"
Oct 30 20:21:06.907131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907112    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652"} err="failed to get container status \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": rpc error: code = NotFound desc = could not find container \"d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652\": container with ID starting with d048228e240aef6e7f5febaeb17cc7b300f6f6c719bce88ed9e7c178f40d5652 not found: ID does not exist"
Oct 30 20:21:06.907174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907133    5967 scope.go:122] "RemoveContainer" containerID="2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"
Oct 30 20:21:06.907339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907320    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4"} err="failed to get container status \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": rpc error: code = NotFound desc = could not find container \"2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4\": container with ID starting with 2551de7a4b148edb98702e878c0595fd0b76f4fe1b2cbaa112f67cc8aa26a1a4 not found: ID does not exist"
Oct 30 20:21:06.907413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907340    5967 scope.go:122] "RemoveContainer" containerID="e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"
Oct 30 20:21:06.907591 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907573    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622"} err="failed to get container status \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": rpc error: code = NotFound desc = could not find container \"e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622\": container with ID starting with e00c831fc93cb1e25b3b6279e2a9c1aec6e9efb3ec3fecb936a121d624437622 not found: ID does not exist"
Oct 30 20:21:06.907632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907591    5967 scope.go:122] "RemoveContainer" containerID="b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"
Oct 30 20:21:06.907808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.907789    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524"} err="failed to get container status \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": rpc error: code = NotFound desc = could not find container \"b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524\": container with ID starting with b5af41a689c6877ebaed79559468f41c8f6b3d6122c3a881b37f502ba29d6524 not found: ID does not exist"
Oct 30 20:21:06.993918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.993888    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:21:06.997312 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:06.997289    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:21:07.024181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024154    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:21:07.024445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024433    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="config-reloader"
Oct 30 20:21:07.024481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024447    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="config-reloader"
Oct 30 20:21:07.024481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024458    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy"
Oct 30 20:21:07.024481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024463    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy"
Oct 30 20:21:07.024481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024477    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-thanos"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024483    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-thanos"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024495    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="prometheus"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024500    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="prometheus"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024506    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-web"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024510    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-web"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024518    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="init-config-reloader"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024522    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="init-config-reloader"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024529    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="thanos-sidecar"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024533    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="thanos-sidecar"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024574    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="config-reloader"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024580    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024588    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-thanos"
Oct 30 20:21:07.024588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024594    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="prometheus"
Oct 30 20:21:07.025006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024599    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="kube-rbac-proxy-web"
Oct 30 20:21:07.025006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.024604    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" containerName="thanos-sidecar"
Oct 30 20:21:07.028592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.028574    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.080781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.080730    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:21:07.081404 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.081383    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:21:07.081532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.081491    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:21:07.081586 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.081542    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:21:07.081801 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.081779    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:21:07.082277 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.082257    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:21:07.082526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.082510    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:21:07.082585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.082558    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:21:07.082782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.082771    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ophv587cgj3g\""
Oct 30 20:21:07.083016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083001    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:21:07.083333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083320    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:21:07.083486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083472    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:21:07.083555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083489    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:21:07.083555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083533    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:21:07.083555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083531    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tlxf9\""
Oct 30 20:21:07.083964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.083949    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:21:07.115140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115111    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115145    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115162    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-config\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115179    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115324    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115342    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d83498-1a98-4352-81f1-2edb48492715-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115359    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115374    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115389    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115451    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-web-config\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115529    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115592    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115624    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115683    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115715    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115735    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpb4\" (UniqueName: \"kubernetes.io/projected/05d83498-1a98-4352-81f1-2edb48492715-kube-api-access-2xpb4\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115802    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d83498-1a98-4352-81f1-2edb48492715-config-out\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115855    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115912    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.115953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.115934    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.133268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.133244    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:21:07.133431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.133416    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:21:07.216413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216384    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216421    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216442    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d83498-1a98-4352-81f1-2edb48492715-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216551    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216585    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216604    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216712    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-web-config\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216775    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216810    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216851    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.216940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216909    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.217192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216958    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.217192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.216995    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpb4\" (UniqueName: \"kubernetes.io/projected/05d83498-1a98-4352-81f1-2edb48492715-kube-api-access-2xpb4\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.217192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.217048    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d83498-1a98-4352-81f1-2edb48492715-config-out\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.217192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.217076    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.219133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.217630    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.219133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.217797    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.219133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.217830    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.219133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.219081    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.219688    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d83498-1a98-4352-81f1-2edb48492715-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.219759    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.219992    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.220311    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.220422    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.220474    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-config\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.221518    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.221687    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.221759    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.221783    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.222327    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.222421    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.222815    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d83498-1a98-4352-81f1-2edb48492715-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.222939    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.224769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.224339    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-web-config\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.226370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.226347    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.226997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.226835    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d83498-1a98-4352-81f1-2edb48492715-config-out\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.227568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.227550    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.227919 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.227902    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-config\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.228567 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.228552    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/05d83498-1a98-4352-81f1-2edb48492715-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.279244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.279182    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpb4\" (UniqueName: \"kubernetes.io/projected/05d83498-1a98-4352-81f1-2edb48492715-kube-api-access-2xpb4\") pod \"prometheus-k8s-0\" (UID: \"05d83498-1a98-4352-81f1-2edb48492715\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.346455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.346419    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:07.539996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.539921    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa8d11e-1a70-4b91-8270-dd8c93d8511c" path="/var/lib/kubelet/pods/8fa8d11e-1a70-4b91-8270-dd8c93d8511c/volumes"
Oct 30 20:21:07.788678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.788644    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:21:07.791621 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:21:07.791566    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d83498_1a98_4352_81f1_2edb48492715.slice/crio-faa991bf20b45174d85ba288967bcca30e34b1daf042d2aa82d95f68964ddf68 WatchSource:0}: Error finding container faa991bf20b45174d85ba288967bcca30e34b1daf042d2aa82d95f68964ddf68: Status 404 returned error can't find the container with id faa991bf20b45174d85ba288967bcca30e34b1daf042d2aa82d95f68964ddf68
Oct 30 20:21:07.832312 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:07.832285    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"faa991bf20b45174d85ba288967bcca30e34b1daf042d2aa82d95f68964ddf68"}
Oct 30 20:21:08.838942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:08.838905    5967 generic.go:356] "Generic (PLEG): container finished" podID="05d83498-1a98-4352-81f1-2edb48492715" containerID="b8f6b0147393077c4abeda3f6591c38addf8370dfdc996def61bde6b49bd9ce2" exitCode=0
Oct 30 20:21:08.839322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:08.838951    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerDied","Data":"b8f6b0147393077c4abeda3f6591c38addf8370dfdc996def61bde6b49bd9ce2"}
Oct 30 20:21:09.382943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.382867    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-595cc465d8-g524b"]
Oct 30 20:21:09.385066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.385038    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.440861    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.440985    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.441023    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.441094    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.441224    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.441299    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.441368    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.441560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.441433    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gft\" (UniqueName: \"kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.445636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.445603    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595cc465d8-g524b"]
Oct 30 20:21:09.542077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542051    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542086    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542211    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542368 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542248    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gft\" (UniqueName: \"kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542368 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542272    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542368 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542287    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542368 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542317    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542368 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542343    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.542602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.542553    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.543370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.543342    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.543924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.543527    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.544027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.543979    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.544566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.544286    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.545226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.545195    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.549527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.549505    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.601460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.601426    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gft\" (UniqueName: \"kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft\") pod \"console-595cc465d8-g524b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") " pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.695692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.695600    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:09.846114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.846080    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"7543df1dce78ad0770de9e05bd964d27deb33a7cb787a4ec3d787a280fc0f5e6"}
Oct 30 20:21:09.846478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.846165    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"2e3f6bcaccb20ce6c26fe4d03b7e8fcb387c25020928cbacc672b303e1ced930"}
Oct 30 20:21:09.846478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.846192    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"e2220fd4ea8053086d24f340f9b3ce3f7ebeea5531b3fbd7eccaf7ec26c854d3"}
Oct 30 20:21:09.846478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.846215    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"b1d2398874565fa9efed63244a47e659a2e68e49ecba7624c4fa7b3ed0b936a5"}
Oct 30 20:21:09.846478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.846237    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"64d2445725b173f0ddd66cd9270f3dc439102c0a1d818532979581a7aabc8577"}
Oct 30 20:21:09.846478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.846259    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"05d83498-1a98-4352-81f1-2edb48492715","Type":"ContainerStarted","Data":"e6ac5510beb2e46239a4e38bebca96f42a9202d9fb0d8156fbaf10219706d2f1"}
Oct 30 20:21:09.980506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:09.980453    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.963447981 podStartE2EDuration="3.980437611s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.98963ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:09.978582441 +0000 UTC m=+50.579384403" watchObservedRunningTime="2026-10-30 20:21:09.980437611 +0000 UTC m=+50.581239550"
Oct 30 20:21:10.143319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:10.143288    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595cc465d8-g524b"]
Oct 30 20:21:10.147124 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:21:10.147099    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9337845a_e9eb_4ccc_90ca_0c88a3ea741b.slice/crio-c7da9a99e7a0069593d1c09995046267458f7a48e68dd981e1b1525d469e30b8 WatchSource:0}: Error finding container c7da9a99e7a0069593d1c09995046267458f7a48e68dd981e1b1525d469e30b8: Status 404 returned error can't find the container with id c7da9a99e7a0069593d1c09995046267458f7a48e68dd981e1b1525d469e30b8
Oct 30 20:21:10.858051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:10.858015    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cc465d8-g524b" event={"ID":"9337845a-e9eb-4ccc-90ca-0c88a3ea741b","Type":"ContainerStarted","Data":"b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45"}
Oct 30 20:21:10.858496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:10.858067    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cc465d8-g524b" event={"ID":"9337845a-e9eb-4ccc-90ca-0c88a3ea741b","Type":"ContainerStarted","Data":"c7da9a99e7a0069593d1c09995046267458f7a48e68dd981e1b1525d469e30b8"}
Oct 30 20:21:10.976105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:10.976058    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-595cc465d8-g524b" podStartSLOduration=1.976044642 podStartE2EDuration="1.976044642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:10.975037983 +0000 UTC m=+51.575839943" watchObservedRunningTime="2026-10-30 20:21:10.976044642 +0000 UTC m=+51.576846581"
Oct 30 20:21:12.346609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:12.346571    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:19.695714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:19.695684    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:19.696117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:19.695830    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:19.701299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:19.701272    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:19.901810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:19.901716    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:21:20.047028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:20.046996    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5466579cfb-k26jn"]
Oct 30 20:21:31.978929 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:31.978889    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:21:45.068690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.068646    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5466579cfb-k26jn" podUID="3d2eb885-9801-45fa-ba06-681db385801e" containerName="console" containerID="cri-o://898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a" gracePeriod=15
Oct 30 20:21:45.322118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.322057    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:21:45.469811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469777    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469826    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469852    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469868    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469891    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469924    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469959    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.469997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.469988    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") pod \"3d2eb885-9801-45fa-ba06-681db385801e\" (UID: \"3d2eb885-9801-45fa-ba06-681db385801e\") "
Oct 30 20:21:45.470340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.470176    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.470438 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.470396    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:45.470438 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.470416    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:45.470587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.470442    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:45.470587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.470555    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:45.472370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.472345    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:45.472445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.472371    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "kube-api-access-q2z4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:45.472445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.472390    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config" pod "3d2eb885-9801-45fa-ba06-681db385801e" (UID: "3d2eb885-9801-45fa-ba06-681db385801e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:45.570810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570783    5967 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570805    5967 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-console-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570814    5967 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d2eb885-9801-45fa-ba06-681db385801e-tmp\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570824    5967 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-oauth-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570833    5967 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-oauth-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570842    5967 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2eb885-9801-45fa-ba06-681db385801e-trusted-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570850    5967 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2eb885-9801-45fa-ba06-681db385801e-console-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.570993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.570858    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2z4d\" (UniqueName: \"kubernetes.io/projected/3d2eb885-9801-45fa-ba06-681db385801e-kube-api-access-q2z4d\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.992839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.992806    5967 generic.go:356] "Generic (PLEG): container finished" podID="3d2eb885-9801-45fa-ba06-681db385801e" containerID="898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a" exitCode=0
Oct 30 20:21:45.992992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.992868    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5466579cfb-k26jn"
Oct 30 20:21:45.992992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.992899    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5466579cfb-k26jn" event={"ID":"3d2eb885-9801-45fa-ba06-681db385801e","Type":"ContainerDied","Data":"898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a"}
Oct 30 20:21:45.992992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.992986    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5466579cfb-k26jn" event={"ID":"3d2eb885-9801-45fa-ba06-681db385801e","Type":"ContainerDied","Data":"c57d7fa9bb0c83dabd471e33410058cac3a6f31dd0cc78a76242620c6cdb9eaf"}
Oct 30 20:21:45.993110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:45.993015    5967 scope.go:122] "RemoveContainer" containerID="898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a"
Oct 30 20:21:46.002801 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:46.002784    5967 scope.go:122] "RemoveContainer" containerID="898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a"
Oct 30 20:21:46.003102 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:21:46.003084    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a\": container with ID starting with 898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a not found: ID does not exist" containerID="898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a"
Oct 30 20:21:46.003187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:46.003109    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a"} err="failed to get container status \"898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a\": rpc error: code = NotFound desc = could not find container \"898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a\": container with ID starting with 898579afc06cf0890840927805a015e135eb2a8df5aa81d9aaf822ed0f36f29a not found: ID does not exist"
Oct 30 20:21:46.154413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:46.154385    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5466579cfb-k26jn"]
Oct 30 20:21:46.203185 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:46.203162    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5466579cfb-k26jn"]
Oct 30 20:21:47.540403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:47.540367    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2eb885-9801-45fa-ba06-681db385801e" path="/var/lib/kubelet/pods/3d2eb885-9801-45fa-ba06-681db385801e/volumes"
Oct 30 20:21:59.931029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:59.930941    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:21:59.946979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:21:59.946954    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:22:00.061051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:00.061023    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:22:01.978698 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:22:01.978659    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:22:07.506175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.506136    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5955665854-mjtts"]
Oct 30 20:22:07.506831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.506630    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3d2eb885-9801-45fa-ba06-681db385801e" containerName="console"
Oct 30 20:22:07.506831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.506651    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2eb885-9801-45fa-ba06-681db385801e" containerName="console"
Oct 30 20:22:07.506831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.506782    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2eb885-9801-45fa-ba06-681db385801e" containerName="console"
Oct 30 20:22:07.508904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.508882    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.567583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.567548    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5955665854-mjtts"]
Oct 30 20:22:07.651313 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651268    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-oauth-serving-cert\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651313 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651307    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwh8\" (UniqueName: \"kubernetes.io/projected/3e87fa39-d194-4728-aba9-91aaa90febd2-kube-api-access-bzwh8\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651332    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e87fa39-d194-4728-aba9-91aaa90febd2-console-serving-cert\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651351    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-service-ca\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651456    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-console-config\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651518    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e87fa39-d194-4728-aba9-91aaa90febd2-console-oauth-config\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651535    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-trusted-ca-bundle\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.651575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.651557    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e87fa39-d194-4728-aba9-91aaa90febd2-tmp\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752273    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e87fa39-d194-4728-aba9-91aaa90febd2-tmp\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752327    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-oauth-serving-cert\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752348    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwh8\" (UniqueName: \"kubernetes.io/projected/3e87fa39-d194-4728-aba9-91aaa90febd2-kube-api-access-bzwh8\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752367    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e87fa39-d194-4728-aba9-91aaa90febd2-console-serving-cert\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752381    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-service-ca\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752401    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-console-config\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752440    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e87fa39-d194-4728-aba9-91aaa90febd2-console-oauth-config\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752459    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-trusted-ca-bundle\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.752803 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.752777    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e87fa39-d194-4728-aba9-91aaa90febd2-tmp\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.753182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.753156    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-oauth-serving-cert\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.753306 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.753156    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-console-config\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.753354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.753321    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-service-ca\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.753482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.753461    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e87fa39-d194-4728-aba9-91aaa90febd2-trusted-ca-bundle\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.754956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.754933    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e87fa39-d194-4728-aba9-91aaa90febd2-console-serving-cert\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.755049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.754971    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e87fa39-d194-4728-aba9-91aaa90febd2-console-oauth-config\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.809237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.809160    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwh8\" (UniqueName: \"kubernetes.io/projected/3e87fa39-d194-4728-aba9-91aaa90febd2-kube-api-access-bzwh8\") pod \"console-5955665854-mjtts\" (UID: \"3e87fa39-d194-4728-aba9-91aaa90febd2\") " pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:07.819046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:07.819020    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:08.186797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:08.186718    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5955665854-mjtts"]
Oct 30 20:22:08.189559 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:22:08.189529    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e87fa39_d194_4728_aba9_91aaa90febd2.slice/crio-b70ad3e5a0efbc858eb35c9cfc296141dc5cac4c5f8c583d1960c45711218c33 WatchSource:0}: Error finding container b70ad3e5a0efbc858eb35c9cfc296141dc5cac4c5f8c583d1960c45711218c33: Status 404 returned error can't find the container with id b70ad3e5a0efbc858eb35c9cfc296141dc5cac4c5f8c583d1960c45711218c33
Oct 30 20:22:09.080059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:09.080025    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5955665854-mjtts" event={"ID":"3e87fa39-d194-4728-aba9-91aaa90febd2","Type":"ContainerStarted","Data":"6d87264f9c18716e441aea971f84ffd5f69ac942abd0bffa4d1ffe7355e96af7"}
Oct 30 20:22:09.080521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:09.080075    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5955665854-mjtts" event={"ID":"3e87fa39-d194-4728-aba9-91aaa90febd2","Type":"ContainerStarted","Data":"b70ad3e5a0efbc858eb35c9cfc296141dc5cac4c5f8c583d1960c45711218c33"}
Oct 30 20:22:09.196782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:09.196453    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5955665854-mjtts" podStartSLOduration=2.196438875 podStartE2EDuration="2.196438875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:09.196277787 +0000 UTC m=+109.797079748" watchObservedRunningTime="2026-10-30 20:22:09.196438875 +0000 UTC m=+109.797240836"
Oct 30 20:22:17.819703 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:17.819660    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:17.820179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:17.819782    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:17.824380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:17.824359    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:18.120879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:18.120794    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5955665854-mjtts"
Oct 30 20:22:18.261396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:18.261369    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595cc465d8-g524b"]
Oct 30 20:22:31.978028 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:22:31.977986    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:22:43.280533 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.280479    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-595cc465d8-g524b" podUID="9337845a-e9eb-4ccc-90ca-0c88a3ea741b" containerName="console" containerID="cri-o://b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45" gracePeriod=15
Oct 30 20:22:43.518117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.518090    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:22:43.568591 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568535    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568591 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568582    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568618    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568665    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568721    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft\" (UniqueName: \"kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568771    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568844    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.568982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568876    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle\") pod \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\" (UID: \"9337845a-e9eb-4ccc-90ca-0c88a3ea741b\") "
Oct 30 20:22:43.569383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.568961    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:43.569506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.569101    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:43.569506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.569142    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:43.569506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.569202    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:43.569506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.569427    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:43.571163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.571134    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "kube-api-access-g6gft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:43.571250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.571162    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:43.571284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.571268    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config" pod "9337845a-e9eb-4ccc-90ca-0c88a3ea741b" (UID: "9337845a-e9eb-4ccc-90ca-0c88a3ea741b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:43.670275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670246    5967 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-oauth-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670270    5967 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-oauth-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670279    5967 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670288    5967 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-tmp\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670296    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6gft\" (UniqueName: \"kubernetes.io/projected/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-kube-api-access-g6gft\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670306    5967 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-config\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670315    5967 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-console-serving-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:43.670459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:43.670323    5967 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9337845a-e9eb-4ccc-90ca-0c88a3ea741b-trusted-ca-bundle\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:44.210218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.210184    5967 generic.go:356] "Generic (PLEG): container finished" podID="9337845a-e9eb-4ccc-90ca-0c88a3ea741b" containerID="b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45" exitCode=0
Oct 30 20:22:44.210419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.210259    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cc465d8-g524b"
Oct 30 20:22:44.210419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.210264    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cc465d8-g524b" event={"ID":"9337845a-e9eb-4ccc-90ca-0c88a3ea741b","Type":"ContainerDied","Data":"b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45"}
Oct 30 20:22:44.210419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.210315    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cc465d8-g524b" event={"ID":"9337845a-e9eb-4ccc-90ca-0c88a3ea741b","Type":"ContainerDied","Data":"c7da9a99e7a0069593d1c09995046267458f7a48e68dd981e1b1525d469e30b8"}
Oct 30 20:22:44.210419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.210341    5967 scope.go:122] "RemoveContainer" containerID="b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45"
Oct 30 20:22:44.218619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.218605    5967 scope.go:122] "RemoveContainer" containerID="b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45"
Oct 30 20:22:44.218867 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:22:44.218845    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45\": container with ID starting with b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45 not found: ID does not exist" containerID="b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45"
Oct 30 20:22:44.218939 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.218876    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45"} err="failed to get container status \"b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45\": rpc error: code = NotFound desc = could not find container \"b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45\": container with ID starting with b26bcd2018f7283592643a0432d808b5bbe0b45b43d06df2b491aa0e58b8ed45 not found: ID does not exist"
Oct 30 20:22:44.371056 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.371027    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595cc465d8-g524b"]
Oct 30 20:22:44.374217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:44.374196    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-595cc465d8-g524b"]
Oct 30 20:22:45.539916 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:22:45.539879    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9337845a-e9eb-4ccc-90ca-0c88a3ea741b" path="/var/lib/kubelet/pods/9337845a-e9eb-4ccc-90ca-0c88a3ea741b/volumes"
Oct 30 20:23:01.979409 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:23:01.979372    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:23:31.979890 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:23:31.979802    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:24:01.978057 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:24:01.978020    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:24:31.978466 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:24:31.978421    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:25:01.977960 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:25:01.977879    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:25:03.868449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.868422    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"]
Oct 30 20:25:03.868855 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.868712    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9337845a-e9eb-4ccc-90ca-0c88a3ea741b" containerName="console"
Oct 30 20:25:03.868855 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.868722    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9337845a-e9eb-4ccc-90ca-0c88a3ea741b" containerName="console"
Oct 30 20:25:03.868855 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.868830    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9337845a-e9eb-4ccc-90ca-0c88a3ea741b" containerName="console"
Oct 30 20:25:03.871512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.871497    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:03.968226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.968198    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhl6n\" (UniqueName: \"kubernetes.io/projected/18f964ed-72f4-42e6-b80e-7cae368f17a4-kube-api-access-mhl6n\") pod \"jobset-operator-674cd59dc5-568sm\" (UID: \"18f964ed-72f4-42e6-b80e-7cae368f17a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:03.968380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.968234    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18f964ed-72f4-42e6-b80e-7cae368f17a4-tmp\") pod \"jobset-operator-674cd59dc5-568sm\" (UID: \"18f964ed-72f4-42e6-b80e-7cae368f17a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:03.972463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.972442    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-nkh7m\""
Oct 30 20:25:03.972463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.972459    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:25:03.972606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.972550    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 20:25:03.973244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:03.973227    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"]
Oct 30 20:25:04.069224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.069194    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhl6n\" (UniqueName: \"kubernetes.io/projected/18f964ed-72f4-42e6-b80e-7cae368f17a4-kube-api-access-mhl6n\") pod \"jobset-operator-674cd59dc5-568sm\" (UID: \"18f964ed-72f4-42e6-b80e-7cae368f17a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:04.069406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.069233    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18f964ed-72f4-42e6-b80e-7cae368f17a4-tmp\") pod \"jobset-operator-674cd59dc5-568sm\" (UID: \"18f964ed-72f4-42e6-b80e-7cae368f17a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:04.069700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.069684    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18f964ed-72f4-42e6-b80e-7cae368f17a4-tmp\") pod \"jobset-operator-674cd59dc5-568sm\" (UID: \"18f964ed-72f4-42e6-b80e-7cae368f17a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:04.125790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.125716    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhl6n\" (UniqueName: \"kubernetes.io/projected/18f964ed-72f4-42e6-b80e-7cae368f17a4-kube-api-access-mhl6n\") pod \"jobset-operator-674cd59dc5-568sm\" (UID: \"18f964ed-72f4-42e6-b80e-7cae368f17a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:04.194375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.194344    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"
Oct 30 20:25:04.563960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.563934    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-568sm"]
Oct 30 20:25:04.566955 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:04.566929    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18f964ed_72f4_42e6_b80e_7cae368f17a4.slice/crio-34a93e0b0bd1031f74f7455a1383f3ad3d20e4416ee80fff46d2595a15ce2edf WatchSource:0}: Error finding container 34a93e0b0bd1031f74f7455a1383f3ad3d20e4416ee80fff46d2595a15ce2edf: Status 404 returned error can't find the container with id 34a93e0b0bd1031f74f7455a1383f3ad3d20e4416ee80fff46d2595a15ce2edf
Oct 30 20:25:04.717460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:04.717431    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm" event={"ID":"18f964ed-72f4-42e6-b80e-7cae368f17a4","Type":"ContainerStarted","Data":"34a93e0b0bd1031f74f7455a1383f3ad3d20e4416ee80fff46d2595a15ce2edf"}
Oct 30 20:25:08.022982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.022951    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"]
Oct 30 20:25:08.033318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.033293    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.072491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.072466    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"]
Oct 30 20:25:08.087686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.087665    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:25:08.087839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.087731    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:25:08.106325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.106297    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.106457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.106332    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.106457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.106361    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.106575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.106499    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.106575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.106531    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqhk\" (UniqueName: \"kubernetes.io/projected/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-kube-api-access-spqhk\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.106678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.106602    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.177791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.177681    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-trbjm\""
Oct 30 20:25:08.177791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.177693    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:25:08.178025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.177886    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:25:08.178252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.178229    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:25:08.207439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.207418    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.207560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.207450    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqhk\" (UniqueName: \"kubernetes.io/projected/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-kube-api-access-spqhk\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.207560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.207496    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.207560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.207553    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.207721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.207581    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.207721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.207620    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.208073 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.208044    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.210133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.210107    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.210440 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.210362    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.210651 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.210625    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.211776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.211733    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.270911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.270879    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqhk\" (UniqueName: \"kubernetes.io/projected/8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d-kube-api-access-spqhk\") pod \"opendatahub-operator-controller-manager-585d4dbb69-9t475\" (UID: \"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.350097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.350029    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:08.735417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.735386    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm" event={"ID":"18f964ed-72f4-42e6-b80e-7cae368f17a4","Type":"ContainerStarted","Data":"8b4b7fe8594b89321b836f4d3e9b60b100ca3618b29277a298dfbb4760ce1e9b"}
Oct 30 20:25:08.860527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.860500    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"]
Oct 30 20:25:08.860933 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:08.860891    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-568sm" podStartSLOduration=1.8467773159999998 podStartE2EDuration="5.860878632s" totalImagesPullingTime="4.014101316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:08.856161787 +0000 UTC m=+289.456963746" watchObservedRunningTime="2026-10-30 20:25:08.860878632 +0000 UTC m=+289.461680602"
Oct 30 20:25:08.861532 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:08.861507    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8920b1da_8b4c_41d0_b6c5_5fdff5b80a0d.slice/crio-97318df90661c9be034143c53897f74dab6c899dccb7c52b64332c86897082b1 WatchSource:0}: Error finding container 97318df90661c9be034143c53897f74dab6c899dccb7c52b64332c86897082b1: Status 404 returned error can't find the container with id 97318df90661c9be034143c53897f74dab6c899dccb7c52b64332c86897082b1
Oct 30 20:25:09.741482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:09.741442    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475" event={"ID":"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d","Type":"ContainerStarted","Data":"97318df90661c9be034143c53897f74dab6c899dccb7c52b64332c86897082b1"}
Oct 30 20:25:12.761280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:12.761249    5967 generic.go:356] "Generic (PLEG): container finished" podID="8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d" containerID="8cfe506523e55935c102187946817549059adc128c39061aedb7405a4788feaa" exitCode=0
Oct 30 20:25:12.761611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:12.761303    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475" event={"ID":"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d","Type":"ContainerDied","Data":"8cfe506523e55935c102187946817549059adc128c39061aedb7405a4788feaa"}
Oct 30 20:25:13.767487 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:13.767444    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475" event={"ID":"8920b1da-8b4c-41d0-b6c5-5fdff5b80a0d","Type":"ContainerStarted","Data":"77d79cc2258ecfde45669faf574cebaca37ceec95931e58ce9dd80b9dd9061ee"}
Oct 30 20:25:13.768009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:13.767561    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:14.052638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:14.052590    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475" podStartSLOduration=2.4980449670000002 podStartE2EDuration="7.052578043s" totalImagesPullingTime="4.068948645s" totalInitContainerRuntime="485.584431ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:14.004009394 +0000 UTC m=+294.604811353" watchObservedRunningTime="2026-10-30 20:25:14.052578043 +0000 UTC m=+294.653379999"
Oct 30 20:25:17.036817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.036774    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"]
Oct 30 20:25:17.040272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.040248    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:17.148600 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.148574    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:25:17.148765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.148600    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wrkfb\""
Oct 30 20:25:17.148765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.148581    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 20:25:17.148765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.148582    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:25:17.149534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.149515    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"]
Oct 30 20:25:17.187532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.187508    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmft9\" (UniqueName: \"kubernetes.io/projected/c4dbcd9d-614f-4f33-8280-2c4e75578ac5-kube-api-access-nmft9\") pod \"dns-operator-controller-manager-8557cbdb84-fnhlj\" (UID: \"c4dbcd9d-614f-4f33-8280-2c4e75578ac5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:17.288010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.287955    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmft9\" (UniqueName: \"kubernetes.io/projected/c4dbcd9d-614f-4f33-8280-2c4e75578ac5-kube-api-access-nmft9\") pod \"dns-operator-controller-manager-8557cbdb84-fnhlj\" (UID: \"c4dbcd9d-614f-4f33-8280-2c4e75578ac5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:17.344606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.344571    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmft9\" (UniqueName: \"kubernetes.io/projected/c4dbcd9d-614f-4f33-8280-2c4e75578ac5-kube-api-access-nmft9\") pod \"dns-operator-controller-manager-8557cbdb84-fnhlj\" (UID: \"c4dbcd9d-614f-4f33-8280-2c4e75578ac5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:17.352756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.352710    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:17.771943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.771915    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"]
Oct 30 20:25:17.775237 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:17.775200    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dbcd9d_614f_4f33_8280_2c4e75578ac5.slice/crio-85d7846c8a8b5e6a70f86bc1e4cd53331a4e19f95a48a40298b65461590989f7 WatchSource:0}: Error finding container 85d7846c8a8b5e6a70f86bc1e4cd53331a4e19f95a48a40298b65461590989f7: Status 404 returned error can't find the container with id 85d7846c8a8b5e6a70f86bc1e4cd53331a4e19f95a48a40298b65461590989f7
Oct 30 20:25:17.783432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:17.783408    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj" event={"ID":"c4dbcd9d-614f-4f33-8280-2c4e75578ac5","Type":"ContainerStarted","Data":"85d7846c8a8b5e6a70f86bc1e4cd53331a4e19f95a48a40298b65461590989f7"}
Oct 30 20:25:18.536295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.536265    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"]
Oct 30 20:25:18.542527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.542510    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.599672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.599428    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"]
Oct 30 20:25:18.613516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.613490    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d3c401-0f49-4263-a5f0-b6ac66470202-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.613614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.613583    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91d3c401-0f49-4263-a5f0-b6ac66470202-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.613665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.613634    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkggd\" (UniqueName: \"kubernetes.io/projected/91d3c401-0f49-4263-a5f0-b6ac66470202-kube-api-access-nkggd\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.644628 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.644605    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 20:25:18.644718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.644662    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 20:25:18.644718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.644690    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rjp9d\""
Oct 30 20:25:18.714488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.714462    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d3c401-0f49-4263-a5f0-b6ac66470202-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.714605 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.714540    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91d3c401-0f49-4263-a5f0-b6ac66470202-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.714605 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.714572    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkggd\" (UniqueName: \"kubernetes.io/projected/91d3c401-0f49-4263-a5f0-b6ac66470202-kube-api-access-nkggd\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.715169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.715151    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91d3c401-0f49-4263-a5f0-b6ac66470202-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.716992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.716966    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d3c401-0f49-4263-a5f0-b6ac66470202-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.771347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.771322    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkggd\" (UniqueName: \"kubernetes.io/projected/91d3c401-0f49-4263-a5f0-b6ac66470202-kube-api-access-nkggd\") pod \"kuadrant-console-plugin-66c4fb575d-9n4jr\" (UID: \"91d3c401-0f49-4263-a5f0-b6ac66470202\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:18.854059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:18.854008    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"
Oct 30 20:25:19.219564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:19.219533    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr"]
Oct 30 20:25:19.222305 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:19.222277    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d3c401_0f49_4263_a5f0_b6ac66470202.slice/crio-c648911a3aceaa7b47de2f9f0fd14b5a9083b0a42be9656aa2644a5d68c64431 WatchSource:0}: Error finding container c648911a3aceaa7b47de2f9f0fd14b5a9083b0a42be9656aa2644a5d68c64431: Status 404 returned error can't find the container with id c648911a3aceaa7b47de2f9f0fd14b5a9083b0a42be9656aa2644a5d68c64431
Oct 30 20:25:19.793584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:19.793533    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr" event={"ID":"91d3c401-0f49-4263-a5f0-b6ac66470202","Type":"ContainerStarted","Data":"c648911a3aceaa7b47de2f9f0fd14b5a9083b0a42be9656aa2644a5d68c64431"}
Oct 30 20:25:21.564922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:21.564893    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 20:25:21.805013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:21.804929    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj" event={"ID":"c4dbcd9d-614f-4f33-8280-2c4e75578ac5","Type":"ContainerStarted","Data":"25dbe5c0e6aca857d5ced210f59e24df644e6f6a8f93861aa4f979782c5121fd"}
Oct 30 20:25:21.805013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:21.805011    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:21.925190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:21.925131    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj" podStartSLOduration=2.188938342 podStartE2EDuration="5.925116199s" totalImagesPullingTime="3.736177857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:21.923923092 +0000 UTC m=+302.524725051" watchObservedRunningTime="2026-10-30 20:25:21.925116199 +0000 UTC m=+302.525918156"
Oct 30 20:25:24.773982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:24.773952    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-9t475"
Oct 30 20:25:25.354579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.354512    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"]
Oct 30 20:25:25.396437 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.396412    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.416372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.416348    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"]
Oct 30 20:25:25.448142 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.448123    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 20:25:25.448232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.448147    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 20:25:25.448232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.448190    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6v8qt\""
Oct 30 20:25:25.477680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.477652    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6ad1b75-89b0-4f75-9d9f-57b629e124d5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-txk25\" (UID: \"e6ad1b75-89b0-4f75-9d9f-57b629e124d5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.477789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.477696    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlns\" (UniqueName: \"kubernetes.io/projected/e6ad1b75-89b0-4f75-9d9f-57b629e124d5-kube-api-access-qqlns\") pod \"cert-manager-webhook-5dbbbf75cc-txk25\" (UID: \"e6ad1b75-89b0-4f75-9d9f-57b629e124d5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.578369 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.578346    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6ad1b75-89b0-4f75-9d9f-57b629e124d5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-txk25\" (UID: \"e6ad1b75-89b0-4f75-9d9f-57b629e124d5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.578463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.578385    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlns\" (UniqueName: \"kubernetes.io/projected/e6ad1b75-89b0-4f75-9d9f-57b629e124d5-kube-api-access-qqlns\") pod \"cert-manager-webhook-5dbbbf75cc-txk25\" (UID: \"e6ad1b75-89b0-4f75-9d9f-57b629e124d5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.635583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.635528    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6ad1b75-89b0-4f75-9d9f-57b629e124d5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-txk25\" (UID: \"e6ad1b75-89b0-4f75-9d9f-57b629e124d5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.635661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.635600    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlns\" (UniqueName: \"kubernetes.io/projected/e6ad1b75-89b0-4f75-9d9f-57b629e124d5-kube-api-access-qqlns\") pod \"cert-manager-webhook-5dbbbf75cc-txk25\" (UID: \"e6ad1b75-89b0-4f75-9d9f-57b629e124d5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.726168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.726148    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:25.833815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.833783    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr" event={"ID":"91d3c401-0f49-4263-a5f0-b6ac66470202","Type":"ContainerStarted","Data":"fd58e17ba53b2ea5a6e45c9176fab38af086f8446dd3afe4d0b6ed4189471f3f"}
Oct 30 20:25:25.950146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:25.950064    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n4jr" podStartSLOduration=2.18632802 podStartE2EDuration="7.950049954s" totalImagesPullingTime="5.763721934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:25.948847311 +0000 UTC m=+306.549649270" watchObservedRunningTime="2026-10-30 20:25:25.950049954 +0000 UTC m=+306.550851913"
Oct 30 20:25:26.111117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:26.111056    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"]
Oct 30 20:25:26.113631 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:26.113603    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6ad1b75_89b0_4f75_9d9f_57b629e124d5.slice/crio-ab04086b3be0ef4c62096999ec8584bb858e50260638772af613e6aef8c8e026 WatchSource:0}: Error finding container ab04086b3be0ef4c62096999ec8584bb858e50260638772af613e6aef8c8e026: Status 404 returned error can't find the container with id ab04086b3be0ef4c62096999ec8584bb858e50260638772af613e6aef8c8e026
Oct 30 20:25:26.115541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:26.115524    5967 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:25:26.839411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:26.839378    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25" event={"ID":"e6ad1b75-89b0-4f75-9d9f-57b629e124d5","Type":"ContainerStarted","Data":"ab04086b3be0ef4c62096999ec8584bb858e50260638772af613e6aef8c8e026"}
Oct 30 20:25:27.119991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.119922    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"]
Oct 30 20:25:27.129098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.129080    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.181135 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.181109    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"]
Oct 30 20:25:27.184171 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.184149    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hqkbf\""
Oct 30 20:25:27.293392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.293367    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2d2d508-6d24-42c1-81de-fe1b16dd5ad2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jj5wm\" (UID: \"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.293497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.293428    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpn2\" (UniqueName: \"kubernetes.io/projected/a2d2d508-6d24-42c1-81de-fe1b16dd5ad2-kube-api-access-7tpn2\") pod \"cert-manager-cainjector-5c57bbd4fc-jj5wm\" (UID: \"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.394106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.394030    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpn2\" (UniqueName: \"kubernetes.io/projected/a2d2d508-6d24-42c1-81de-fe1b16dd5ad2-kube-api-access-7tpn2\") pod \"cert-manager-cainjector-5c57bbd4fc-jj5wm\" (UID: \"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.394106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.394096    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2d2d508-6d24-42c1-81de-fe1b16dd5ad2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jj5wm\" (UID: \"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.450827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.450801    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2d2d508-6d24-42c1-81de-fe1b16dd5ad2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jj5wm\" (UID: \"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.450827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.450823    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpn2\" (UniqueName: \"kubernetes.io/projected/a2d2d508-6d24-42c1-81de-fe1b16dd5ad2-kube-api-access-7tpn2\") pod \"cert-manager-cainjector-5c57bbd4fc-jj5wm\" (UID: \"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:27.739111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:27.739083    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"
Oct 30 20:25:28.113786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:28.113653    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm"]
Oct 30 20:25:28.117840 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:28.117813    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d2d508_6d24_42c1_81de_fe1b16dd5ad2.slice/crio-3148c7bbe85917f993d09dc447c24f87e20c38f0cb04d7f80570851c4dd3a1df WatchSource:0}: Error finding container 3148c7bbe85917f993d09dc447c24f87e20c38f0cb04d7f80570851c4dd3a1df: Status 404 returned error can't find the container with id 3148c7bbe85917f993d09dc447c24f87e20c38f0cb04d7f80570851c4dd3a1df
Oct 30 20:25:28.849936 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:28.849898    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm" event={"ID":"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2","Type":"ContainerStarted","Data":"3148c7bbe85917f993d09dc447c24f87e20c38f0cb04d7f80570851c4dd3a1df"}
Oct 30 20:25:31.866304 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:31.866267    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm" event={"ID":"a2d2d508-6d24-42c1-81de-fe1b16dd5ad2","Type":"ContainerStarted","Data":"579efa348f979ad25a53e227ebf79340238139c36a0e05f37d399bbb5788adb3"}
Oct 30 20:25:31.868027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:31.868000    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25" event={"ID":"e6ad1b75-89b0-4f75-9d9f-57b629e124d5","Type":"ContainerStarted","Data":"25198dd74b07ec43e3334eb0e2ac4688a4919a601daf321be79a708ead00ea30"}
Oct 30 20:25:31.868133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:31.868047    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:31.980422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:31.980364    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jj5wm" podStartSLOduration=2.146514262 podStartE2EDuration="4.980347009s" totalImagesPullingTime="2.833832747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:31.979180905 +0000 UTC m=+312.579982878" watchObservedRunningTime="2026-10-30 20:25:31.980347009 +0000 UTC m=+312.581148971"
Oct 30 20:25:31.980705 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:25:31.980678    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:25:32.093922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:32.093872    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25" podStartSLOduration=2.369972729 podStartE2EDuration="7.093852593s" totalImagesPullingTime="4.723879864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:32.093231008 +0000 UTC m=+312.694032968" watchObservedRunningTime="2026-10-30 20:25:32.093852593 +0000 UTC m=+312.694654556"
Oct 30 20:25:32.811877 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:32.811844    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fnhlj"
Oct 30 20:25:35.044672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.044641    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-kf6dv"]
Oct 30 20:25:35.047198 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.047179    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.060806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.060777    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92e15137-61ed-49d3-bec9-46682259abdd-bound-sa-token\") pod \"cert-manager-597b95cb84-kf6dv\" (UID: \"92e15137-61ed-49d3-bec9-46682259abdd\") " pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.060929 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.060832    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8d5\" (UniqueName: \"kubernetes.io/projected/92e15137-61ed-49d3-bec9-46682259abdd-kube-api-access-bc8d5\") pod \"cert-manager-597b95cb84-kf6dv\" (UID: \"92e15137-61ed-49d3-bec9-46682259abdd\") " pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.148417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.148395    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n45wr\""
Oct 30 20:25:35.149458 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.149439    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kf6dv"]
Oct 30 20:25:35.161541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.161514    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92e15137-61ed-49d3-bec9-46682259abdd-bound-sa-token\") pod \"cert-manager-597b95cb84-kf6dv\" (UID: \"92e15137-61ed-49d3-bec9-46682259abdd\") " pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.161634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.161551    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8d5\" (UniqueName: \"kubernetes.io/projected/92e15137-61ed-49d3-bec9-46682259abdd-kube-api-access-bc8d5\") pod \"cert-manager-597b95cb84-kf6dv\" (UID: \"92e15137-61ed-49d3-bec9-46682259abdd\") " pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.217463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.217434    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92e15137-61ed-49d3-bec9-46682259abdd-bound-sa-token\") pod \"cert-manager-597b95cb84-kf6dv\" (UID: \"92e15137-61ed-49d3-bec9-46682259abdd\") " pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.217618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.217599    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8d5\" (UniqueName: \"kubernetes.io/projected/92e15137-61ed-49d3-bec9-46682259abdd-kube-api-access-bc8d5\") pod \"cert-manager-597b95cb84-kf6dv\" (UID: \"92e15137-61ed-49d3-bec9-46682259abdd\") " pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.357642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.357562    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kf6dv"
Oct 30 20:25:35.721504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.721478    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kf6dv"]
Oct 30 20:25:35.724080 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:35.724052    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92e15137_61ed_49d3_bec9_46682259abdd.slice/crio-d65f93bd8643c2f7b0c3e374e71472c8d946efec64434dd61568b98a152d6da2 WatchSource:0}: Error finding container d65f93bd8643c2f7b0c3e374e71472c8d946efec64434dd61568b98a152d6da2: Status 404 returned error can't find the container with id d65f93bd8643c2f7b0c3e374e71472c8d946efec64434dd61568b98a152d6da2
Oct 30 20:25:35.884890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.884860    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kf6dv" event={"ID":"92e15137-61ed-49d3-bec9-46682259abdd","Type":"ContainerStarted","Data":"7710830b03770659ad1024fa7f6afaee7b1236366c664dcaad6f2289ff7b6813"}
Oct 30 20:25:35.885063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.884899    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kf6dv" event={"ID":"92e15137-61ed-49d3-bec9-46682259abdd","Type":"ContainerStarted","Data":"d65f93bd8643c2f7b0c3e374e71472c8d946efec64434dd61568b98a152d6da2"}
Oct 30 20:25:35.998666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:35.998575    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-kf6dv" podStartSLOduration=0.998562211 podStartE2EDuration="998.562211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:35.998066525 +0000 UTC m=+316.598868484" watchObservedRunningTime="2026-10-30 20:25:35.998562211 +0000 UTC m=+316.599364163"
Oct 30 20:25:37.874920 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:37.874890    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-txk25"
Oct 30 20:25:38.584882 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.584850    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"]
Oct 30 20:25:38.587273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.587252    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.639196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.639173    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:25:38.639196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.639188    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:25:38.639343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.639213    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:25:38.688635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.688611    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8g74l\""
Oct 30 20:25:38.688757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.688637    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 20:25:38.689808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.689791    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"]
Oct 30 20:25:38.693307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.693289    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xw2\" (UniqueName: \"kubernetes.io/projected/493b7e7c-228f-481c-9a88-aa6b798854e5-kube-api-access-s4xw2\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.693365 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.693323    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/493b7e7c-228f-481c-9a88-aa6b798854e5-tmp\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.693365 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.693358    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493b7e7c-228f-481c-9a88-aa6b798854e5-tls-certs\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.794333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.794299    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493b7e7c-228f-481c-9a88-aa6b798854e5-tls-certs\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.794480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.794415    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xw2\" (UniqueName: \"kubernetes.io/projected/493b7e7c-228f-481c-9a88-aa6b798854e5-kube-api-access-s4xw2\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.794480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.794459    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/493b7e7c-228f-481c-9a88-aa6b798854e5-tmp\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.796600 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.796577    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/493b7e7c-228f-481c-9a88-aa6b798854e5-tmp\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.796700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.796624    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493b7e7c-228f-481c-9a88-aa6b798854e5-tls-certs\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.853065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.852995    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xw2\" (UniqueName: \"kubernetes.io/projected/493b7e7c-228f-481c-9a88-aa6b798854e5-kube-api-access-s4xw2\") pod \"kube-auth-proxy-557f979cb4-ctpdv\" (UID: \"493b7e7c-228f-481c-9a88-aa6b798854e5\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:38.898657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:38.898629    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"
Oct 30 20:25:39.269860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.269832    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv"]
Oct 30 20:25:39.273126 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:39.273099    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493b7e7c_228f_481c_9a88_aa6b798854e5.slice/crio-31bdcd41d2d468cb527cb1ef1b1650125a7971e542963103d8a7c1edfdbe82be WatchSource:0}: Error finding container 31bdcd41d2d468cb527cb1ef1b1650125a7971e542963103d8a7c1edfdbe82be: Status 404 returned error can't find the container with id 31bdcd41d2d468cb527cb1ef1b1650125a7971e542963103d8a7c1edfdbe82be
Oct 30 20:25:39.762098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.762071    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h4pxz"]
Oct 30 20:25:39.765344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.765318    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:25:39.803014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.802987    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cf6w\" (UniqueName: \"kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w\") pod \"authorino-57f7f8659f-h4pxz\" (UID: \"45b980fa-08bf-4bde-a752-35d347298ca1\") " pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:25:39.818330 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.818309    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2p4vw\""
Oct 30 20:25:39.827423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.827401    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h4pxz"]
Oct 30 20:25:39.903806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.903771    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cf6w\" (UniqueName: \"kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w\") pod \"authorino-57f7f8659f-h4pxz\" (UID: \"45b980fa-08bf-4bde-a752-35d347298ca1\") " pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:25:39.904955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.904934    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv" event={"ID":"493b7e7c-228f-481c-9a88-aa6b798854e5","Type":"ContainerStarted","Data":"31bdcd41d2d468cb527cb1ef1b1650125a7971e542963103d8a7c1edfdbe82be"}
Oct 30 20:25:39.967724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:39.967691    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cf6w\" (UniqueName: \"kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w\") pod \"authorino-57f7f8659f-h4pxz\" (UID: \"45b980fa-08bf-4bde-a752-35d347298ca1\") " pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:25:40.076935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:40.076857    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:25:40.449164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:40.449087    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h4pxz"]
Oct 30 20:25:40.452925 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:40.452892    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b980fa_08bf_4bde_a752_35d347298ca1.slice/crio-37ef09d8fde968d8a4a0a9bf2a4055a726f18bb4b378953d026b2d2407d4ab03 WatchSource:0}: Error finding container 37ef09d8fde968d8a4a0a9bf2a4055a726f18bb4b378953d026b2d2407d4ab03: Status 404 returned error can't find the container with id 37ef09d8fde968d8a4a0a9bf2a4055a726f18bb4b378953d026b2d2407d4ab03
Oct 30 20:25:40.913108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:40.913059    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h4pxz" event={"ID":"45b980fa-08bf-4bde-a752-35d347298ca1","Type":"ContainerStarted","Data":"37ef09d8fde968d8a4a0a9bf2a4055a726f18bb4b378953d026b2d2407d4ab03"}
Oct 30 20:25:42.925553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:42.925521    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv" event={"ID":"493b7e7c-228f-481c-9a88-aa6b798854e5","Type":"ContainerStarted","Data":"87c88365352d33a532adcb729dbcc93ff39f34c076a9d0d5df9b1b76c571ace0"}
Oct 30 20:25:43.930988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:43.930950    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h4pxz" event={"ID":"45b980fa-08bf-4bde-a752-35d347298ca1","Type":"ContainerStarted","Data":"f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9"}
Oct 30 20:25:44.042492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:44.042443    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-h4pxz" podStartSLOduration=2.234416091 podStartE2EDuration="5.042426613s" totalImagesPullingTime="2.808010522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:44.042204981 +0000 UTC m=+324.643006941" watchObservedRunningTime="2026-10-30 20:25:44.042426613 +0000 UTC m=+324.643228576"
Oct 30 20:25:44.043011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:44.042978    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv" podStartSLOduration=3.189869865 podStartE2EDuration="6.042968906s" totalImagesPullingTime="2.853099041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:43.041176287 +0000 UTC m=+323.641978250" watchObservedRunningTime="2026-10-30 20:25:44.042968906 +0000 UTC m=+324.643770882"
Oct 30 20:25:58.842336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.842298    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"]
Oct 30 20:25:58.860522 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.860499    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:58.903795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.903769    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"]
Oct 30 20:25:58.952117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.952065    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 20:25:58.962184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.962157    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-kwrwp\""
Oct 30 20:25:58.962184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.962163    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 20:25:58.962184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.962163    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 20:25:58.988082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.988057    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fcgx\" (UniqueName: \"kubernetes.io/projected/b4ae59e2-5d51-41cc-81c0-d03716eb5695-kube-api-access-2fcgx\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:58.988196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.988119    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ae59e2-5d51-41cc-81c0-d03716eb5695-metrics-certs\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:58.988259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.988232    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4ae59e2-5d51-41cc-81c0-d03716eb5695-manager-config\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:58.988312 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:58.988282    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ae59e2-5d51-41cc-81c0-d03716eb5695-cert\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.089687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.089654    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ae59e2-5d51-41cc-81c0-d03716eb5695-metrics-certs\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.089852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.089702    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4ae59e2-5d51-41cc-81c0-d03716eb5695-manager-config\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.089852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.089754    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ae59e2-5d51-41cc-81c0-d03716eb5695-cert\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.089852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.089787    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcgx\" (UniqueName: \"kubernetes.io/projected/b4ae59e2-5d51-41cc-81c0-d03716eb5695-kube-api-access-2fcgx\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.090353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.090330    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4ae59e2-5d51-41cc-81c0-d03716eb5695-manager-config\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.092213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.092192    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ae59e2-5d51-41cc-81c0-d03716eb5695-cert\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.092426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.092387    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ae59e2-5d51-41cc-81c0-d03716eb5695-metrics-certs\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.146186 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.146163    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcgx\" (UniqueName: \"kubernetes.io/projected/b4ae59e2-5d51-41cc-81c0-d03716eb5695-kube-api-access-2fcgx\") pod \"jobset-controller-manager-5bf5d44b8b-9s7kz\" (UID: \"b4ae59e2-5d51-41cc-81c0-d03716eb5695\") " pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.172069 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.172050    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:25:59.541342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:25:59.541309    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"]
Oct 30 20:25:59.543444 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:25:59.543420    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ae59e2_5d51_41cc_81c0_d03716eb5695.slice/crio-bc7124cc102787d4f84b64bf4b804d5ff2c2882934e0b19bc9ee7ec2d6832052 WatchSource:0}: Error finding container bc7124cc102787d4f84b64bf4b804d5ff2c2882934e0b19bc9ee7ec2d6832052: Status 404 returned error can't find the container with id bc7124cc102787d4f84b64bf4b804d5ff2c2882934e0b19bc9ee7ec2d6832052
Oct 30 20:26:00.001037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:00.000996    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz" event={"ID":"b4ae59e2-5d51-41cc-81c0-d03716eb5695","Type":"ContainerStarted","Data":"bc7124cc102787d4f84b64bf4b804d5ff2c2882934e0b19bc9ee7ec2d6832052"}
Oct 30 20:26:01.981939 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:26:01.981901    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:26:04.022327 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:04.022293    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz" event={"ID":"b4ae59e2-5d51-41cc-81c0-d03716eb5695","Type":"ContainerStarted","Data":"aa3bb5236d2b39d1bc0035cbf4ff4c2127dc3d85b921e1feb5e38ab05fee57cd"}
Oct 30 20:26:04.022775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:04.022414    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:26:04.138841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:04.138778    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz" podStartSLOduration=2.021661131 podStartE2EDuration="6.138726675s" totalImagesPullingTime="4.117065544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:04.138252543 +0000 UTC m=+344.739054503" watchObservedRunningTime="2026-10-30 20:26:04.138726675 +0000 UTC m=+344.739528636"
Oct 30 20:26:04.996278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:04.996251    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h4pxz"]
Oct 30 20:26:04.996444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:04.996424    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-h4pxz" podUID="45b980fa-08bf-4bde-a752-35d347298ca1" containerName="authorino" containerID="cri-o://f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9" gracePeriod=30
Oct 30 20:26:05.256603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:05.256553    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:26:05.345191 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:05.345165    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w\" (UniqueName: \"kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w\") pod \"45b980fa-08bf-4bde-a752-35d347298ca1\" (UID: \"45b980fa-08bf-4bde-a752-35d347298ca1\") "
Oct 30 20:26:05.347244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:05.347219    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w" pod "45b980fa-08bf-4bde-a752-35d347298ca1" (UID: "45b980fa-08bf-4bde-a752-35d347298ca1"). InnerVolumeSpecName "kube-api-access-2cf6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:05.446284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:05.446253    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cf6w\" (UniqueName: \"kubernetes.io/projected/45b980fa-08bf-4bde-a752-35d347298ca1-kube-api-access-2cf6w\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:06.032561 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.032527    5967 generic.go:356] "Generic (PLEG): container finished" podID="45b980fa-08bf-4bde-a752-35d347298ca1" containerID="f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9" exitCode=0
Oct 30 20:26:06.032712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.032576    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h4pxz"
Oct 30 20:26:06.032712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.032608    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h4pxz" event={"ID":"45b980fa-08bf-4bde-a752-35d347298ca1","Type":"ContainerDied","Data":"f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9"}
Oct 30 20:26:06.032712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.032651    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h4pxz" event={"ID":"45b980fa-08bf-4bde-a752-35d347298ca1","Type":"ContainerDied","Data":"37ef09d8fde968d8a4a0a9bf2a4055a726f18bb4b378953d026b2d2407d4ab03"}
Oct 30 20:26:06.032712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.032674    5967 scope.go:122] "RemoveContainer" containerID="f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9"
Oct 30 20:26:06.040851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.040829    5967 scope.go:122] "RemoveContainer" containerID="f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9"
Oct 30 20:26:06.041101 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:26:06.041083    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9\": container with ID starting with f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9 not found: ID does not exist" containerID="f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9"
Oct 30 20:26:06.041163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.041110    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9"} err="failed to get container status \"f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9\": rpc error: code = NotFound desc = could not find container \"f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9\": container with ID starting with f27644e0fc15375fb358dc47d2dfe7734c35b96c69c1334eefdcc98e853807b9 not found: ID does not exist"
Oct 30 20:26:06.193146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.193121    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h4pxz"]
Oct 30 20:26:06.242504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:06.242476    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h4pxz"]
Oct 30 20:26:07.540190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:07.540158    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b980fa-08bf-4bde-a752-35d347298ca1" path="/var/lib/kubelet/pods/45b980fa-08bf-4bde-a752-35d347298ca1/volumes"
Oct 30 20:26:15.031376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:15.031345    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5bf5d44b8b-9s7kz"
Oct 30 20:26:31.979275 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:26:31.979200    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:26:57.002294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.002259    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2fpxj"]
Oct 30 20:26:57.002835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.002817    5967 cpu_manager.go:414] "containerMap: removing container" podUID="45b980fa-08bf-4bde-a752-35d347298ca1" containerName="authorino"
Oct 30 20:26:57.002930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.002836    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b980fa-08bf-4bde-a752-35d347298ca1" containerName="authorino"
Oct 30 20:26:57.002983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.002943    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b980fa-08bf-4bde-a752-35d347298ca1" containerName="authorino"
Oct 30 20:26:57.007430 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.007408    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.061340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.061318    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:26:57.085467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.085436    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785df042-e446-406f-87d7-2325e753dbc0-tls-certs\") pod \"model-serving-api-75bd89fdfc-2fpxj\" (UID: \"785df042-e446-406f-87d7-2325e753dbc0\") " pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.085602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.085503    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b65x\" (UniqueName: \"kubernetes.io/projected/785df042-e446-406f-87d7-2325e753dbc0-kube-api-access-9b65x\") pod \"model-serving-api-75bd89fdfc-2fpxj\" (UID: \"785df042-e446-406f-87d7-2325e753dbc0\") " pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.110839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.110807    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:26:57.110961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.110884    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 20:26:57.110961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.110937    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ndmnx\""
Oct 30 20:26:57.112351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.112288    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2fpxj"]
Oct 30 20:26:57.113326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.113308    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-6hgn6"]
Oct 30 20:26:57.115707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.115692    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.170755 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.170721    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t4t5x\""
Oct 30 20:26:57.170897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.170831    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 20:26:57.170897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.170869    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 20:26:57.175115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.175094    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6hgn6"]
Oct 30 20:26:57.186269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.186240    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785df042-e446-406f-87d7-2325e753dbc0-tls-certs\") pod \"model-serving-api-75bd89fdfc-2fpxj\" (UID: \"785df042-e446-406f-87d7-2325e753dbc0\") " pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.186373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.186300    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrbn\" (UniqueName: \"kubernetes.io/projected/a3057d5f-f9ff-4941-bca4-15efadba993e-kube-api-access-lbrbn\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.186373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.186362    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3057d5f-f9ff-4941-bca4-15efadba993e-metrics-certs\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.186482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.186400    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b65x\" (UniqueName: \"kubernetes.io/projected/785df042-e446-406f-87d7-2325e753dbc0-kube-api-access-9b65x\") pod \"model-serving-api-75bd89fdfc-2fpxj\" (UID: \"785df042-e446-406f-87d7-2325e753dbc0\") " pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.186673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.186639    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3057d5f-f9ff-4941-bca4-15efadba993e-cert\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.188840 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.188825    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785df042-e446-406f-87d7-2325e753dbc0-tls-certs\") pod \"model-serving-api-75bd89fdfc-2fpxj\" (UID: \"785df042-e446-406f-87d7-2325e753dbc0\") " pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.243408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.243385    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b65x\" (UniqueName: \"kubernetes.io/projected/785df042-e446-406f-87d7-2325e753dbc0-kube-api-access-9b65x\") pod \"model-serving-api-75bd89fdfc-2fpxj\" (UID: \"785df042-e446-406f-87d7-2325e753dbc0\") " pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.287202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.287149    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrbn\" (UniqueName: \"kubernetes.io/projected/a3057d5f-f9ff-4941-bca4-15efadba993e-kube-api-access-lbrbn\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.287288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.287206    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3057d5f-f9ff-4941-bca4-15efadba993e-metrics-certs\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.287288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.287251    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3057d5f-f9ff-4941-bca4-15efadba993e-cert\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.289644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.289627    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3057d5f-f9ff-4941-bca4-15efadba993e-cert\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.289880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.289859    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3057d5f-f9ff-4941-bca4-15efadba993e-metrics-certs\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.318555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.318536    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:26:57.343828 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.343803    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrbn\" (UniqueName: \"kubernetes.io/projected/a3057d5f-f9ff-4941-bca4-15efadba993e-kube-api-access-lbrbn\") pod \"odh-model-controller-769875db56-6hgn6\" (UID: \"a3057d5f-f9ff-4941-bca4-15efadba993e\") " pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.427411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.427380    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:26:57.683105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.682904    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2fpxj"]
Oct 30 20:26:57.685795 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:26:57.685768    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785df042_e446_406f_87d7_2325e753dbc0.slice/crio-9ba8f34b45861e0dd7051cc22d1de53d3dfa0c04373381f0271062c0741f85af WatchSource:0}: Error finding container 9ba8f34b45861e0dd7051cc22d1de53d3dfa0c04373381f0271062c0741f85af: Status 404 returned error can't find the container with id 9ba8f34b45861e0dd7051cc22d1de53d3dfa0c04373381f0271062c0741f85af
Oct 30 20:26:57.793362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:57.793341    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6hgn6"]
Oct 30 20:26:57.795755 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:26:57.795708    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3057d5f_f9ff_4941_bca4_15efadba993e.slice/crio-f9f5ae0be5610ed180aaeb0f3d15a8fd98546738977d344903d8e9f19979e767 WatchSource:0}: Error finding container f9f5ae0be5610ed180aaeb0f3d15a8fd98546738977d344903d8e9f19979e767: Status 404 returned error can't find the container with id f9f5ae0be5610ed180aaeb0f3d15a8fd98546738977d344903d8e9f19979e767
Oct 30 20:26:58.261084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:58.261050    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj" event={"ID":"785df042-e446-406f-87d7-2325e753dbc0","Type":"ContainerStarted","Data":"9ba8f34b45861e0dd7051cc22d1de53d3dfa0c04373381f0271062c0741f85af"}
Oct 30 20:26:58.262312 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:26:58.262289    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6hgn6" event={"ID":"a3057d5f-f9ff-4941-bca4-15efadba993e","Type":"ContainerStarted","Data":"f9f5ae0be5610ed180aaeb0f3d15a8fd98546738977d344903d8e9f19979e767"}
Oct 30 20:27:00.275335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:00.275295    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj" event={"ID":"785df042-e446-406f-87d7-2325e753dbc0","Type":"ContainerStarted","Data":"77f89555856a0879d7bf43f0ad481f7534811a8d459b3505ae30b7a82200a171"}
Oct 30 20:27:00.275796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:00.275589    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:27:00.393389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:00.393286    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj" podStartSLOduration=1.9947866950000002 podStartE2EDuration="4.393272193s" totalImagesPullingTime="2.398485498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:00.392206986 +0000 UTC m=+400.993008946" watchObservedRunningTime="2026-10-30 20:27:00.393272193 +0000 UTC m=+400.994074149"
Oct 30 20:27:01.281551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:01.281464    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6hgn6" event={"ID":"a3057d5f-f9ff-4941-bca4-15efadba993e","Type":"ContainerStarted","Data":"b9040befa3ff525906e1884a8ece8856b00432f8fc2a76cd78d4a08261ddec21"}
Oct 30 20:27:01.281934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:01.281732    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:27:01.399637 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:01.399588    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-6hgn6" podStartSLOduration=1.210444126 podStartE2EDuration="4.399573524s" totalImagesPullingTime="3.189129398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:01.397517023 +0000 UTC m=+401.998318982" watchObservedRunningTime="2026-10-30 20:27:01.399573524 +0000 UTC m=+402.000375483"
Oct 30 20:27:01.978660 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:27:01.978624    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:27:11.286003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:11.285973    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-2fpxj"
Oct 30 20:27:12.288589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:12.288559    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-6hgn6"
Oct 30 20:27:17.834921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.834890    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-745f99f494-cphzg"]
Oct 30 20:27:17.837224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.837207    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:17.868756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.868721    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert\") pod \"authorino-745f99f494-cphzg\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") " pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:17.868898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.868877    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkf8\" (UniqueName: \"kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8\") pod \"authorino-745f99f494-cphzg\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") " pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:17.941494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.941472    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2p4vw\""
Oct 30 20:27:17.941494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.941488    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 20:27:17.942626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.942610    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-745f99f494-cphzg"]
Oct 30 20:27:17.969789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.969768    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkf8\" (UniqueName: \"kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8\") pod \"authorino-745f99f494-cphzg\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") " pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:17.969873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.969815    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert\") pod \"authorino-745f99f494-cphzg\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") " pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:17.972237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:17.972219    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert\") pod \"authorino-745f99f494-cphzg\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") " pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:18.027192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:18.027171    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkf8\" (UniqueName: \"kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8\") pod \"authorino-745f99f494-cphzg\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") " pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:18.148031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:18.147961    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:27:18.521575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:18.521547    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-745f99f494-cphzg"]
Oct 30 20:27:18.525177 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:27:18.525150    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5f7ca12_d2d6_4a41_8396_1f635f9d8f18.slice/crio-b98c375d28c04aebf924b63dd9691dce99d63ce6e466bc033ffed8fabfb30dec WatchSource:0}: Error finding container b98c375d28c04aebf924b63dd9691dce99d63ce6e466bc033ffed8fabfb30dec: Status 404 returned error can't find the container with id b98c375d28c04aebf924b63dd9691dce99d63ce6e466bc033ffed8fabfb30dec
Oct 30 20:27:19.361233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:19.361198    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-745f99f494-cphzg" event={"ID":"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18","Type":"ContainerStarted","Data":"b98c375d28c04aebf924b63dd9691dce99d63ce6e466bc033ffed8fabfb30dec"}
Oct 30 20:27:20.367040 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:20.367004    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-745f99f494-cphzg" event={"ID":"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18","Type":"ContainerStarted","Data":"81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177"}
Oct 30 20:27:20.490235 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:27:20.490191    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-745f99f494-cphzg" podStartSLOduration=2.067603548 podStartE2EDuration="3.490178807s" totalImagesPullingTime="1.422575259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:20.489317724 +0000 UTC m=+421.090119685" watchObservedRunningTime="2026-10-30 20:27:20.490178807 +0000 UTC m=+421.090980766"
Oct 30 20:27:31.978851 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:27:31.978818    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:28:01.978976 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:28:01.978903    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:28:31.978294 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:28:31.978256    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:29:01.978107 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:29:01.978073    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:29:30.806122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.806044    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"]
Oct 30 20:29:30.808909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.808886    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.860959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.860930    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 20:29:30.860959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.860941    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 20:29:30.899072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.899042    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd69661d-c5d4-446c-ad59-866e2e47adc7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.899231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.899087    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.899231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.899154    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.899231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.899193    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxzpx\" (UniqueName: \"kubernetes.io/projected/fd69661d-c5d4-446c-ad59-866e2e47adc7-kube-api-access-zxzpx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.899344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.899285    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.899344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.899315    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:30.910323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.910299    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m9jq4\""
Oct 30 20:29:30.910492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.910385    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 20:29:30.911727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.911703    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"]
Oct 30 20:29:30.999973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:30.999941    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxzpx\" (UniqueName: \"kubernetes.io/projected/fd69661d-c5d4-446c-ad59-866e2e47adc7-kube-api-access-zxzpx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000003    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000037    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000097    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd69661d-c5d4-446c-ad59-866e2e47adc7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000151    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000191    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000414    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000521    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.000602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.000574    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.002518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.002497    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd69661d-c5d4-446c-ad59-866e2e47adc7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.002791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.002772    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd69661d-c5d4-446c-ad59-866e2e47adc7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.056266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.056184    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxzpx\" (UniqueName: \"kubernetes.io/projected/fd69661d-c5d4-446c-ad59-866e2e47adc7-kube-api-access-zxzpx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8\" (UID: \"fd69661d-c5d4-446c-ad59-866e2e47adc7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.118670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.118639    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:31.490724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.490680    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"]
Oct 30 20:29:31.494083 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:29:31.494052    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd69661d_c5d4_446c_ad59_866e2e47adc7.slice/crio-f3edb64370955333f0d626bd1409574d74abbd181a43f18a956a15831717d2b7 WatchSource:0}: Error finding container f3edb64370955333f0d626bd1409574d74abbd181a43f18a956a15831717d2b7: Status 404 returned error can't find the container with id f3edb64370955333f0d626bd1409574d74abbd181a43f18a956a15831717d2b7
Oct 30 20:29:31.945137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:31.945054    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" event={"ID":"fd69661d-c5d4-446c-ad59-866e2e47adc7","Type":"ContainerStarted","Data":"f3edb64370955333f0d626bd1409574d74abbd181a43f18a956a15831717d2b7"}
Oct 30 20:29:31.979111 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:29:31.979079    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:29:33.955447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:33.955415    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" event={"ID":"fd69661d-c5d4-446c-ad59-866e2e47adc7","Type":"ContainerStarted","Data":"96728a588a203e696c0563509289888f85618683b747473729a0d2f928135656"}
Oct 30 20:29:33.965680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:33.965654    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"]
Oct 30 20:29:33.968937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:33.968923    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.020565 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.020544    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 20:29:34.058021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.057998    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"]
Oct 30 20:29:34.078427 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.078386    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podStartSLOduration=2.601689984 podStartE2EDuration="4.078374804s" totalImagesPullingTime="1.47668482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:34.077332936 +0000 UTC m=+554.678134898" watchObservedRunningTime="2026-10-30 20:29:34.078374804 +0000 UTC m=+554.679176763"
Oct 30 20:29:34.119455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.119432    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:34.132355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.132331    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d876e30-bfc0-4db6-a136-e13e456d3abc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.132446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.132359    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptll\" (UniqueName: \"kubernetes.io/projected/5d876e30-bfc0-4db6-a136-e13e456d3abc-kube-api-access-kptll\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.132498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.132460    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.132534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.132501    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.132580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.132556    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.132679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.132665    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233364 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233339    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233370    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d876e30-bfc0-4db6-a136-e13e456d3abc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233387    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptll\" (UniqueName: \"kubernetes.io/projected/5d876e30-bfc0-4db6-a136-e13e456d3abc-kube-api-access-kptll\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233414    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233432    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233476    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233786    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233888 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233844    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.233926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.233895    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.235690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.235675    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d876e30-bfc0-4db6-a136-e13e456d3abc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.235975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.235960    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d876e30-bfc0-4db6-a136-e13e456d3abc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.289033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.289010    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptll\" (UniqueName: \"kubernetes.io/projected/5d876e30-bfc0-4db6-a136-e13e456d3abc-kube-api-access-kptll\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh\" (UID: \"5d876e30-bfc0-4db6-a136-e13e456d3abc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.580807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.580706    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:34.962053 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:34.961986    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"]
Oct 30 20:29:34.965197 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:29:34.965164    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d876e30_bfc0_4db6_a136_e13e456d3abc.slice/crio-4fc80c9e4ebc41e5cd8f464f4d01f94135fdfd2ef8734baf7bcdabcbdccb1ab7 WatchSource:0}: Error finding container 4fc80c9e4ebc41e5cd8f464f4d01f94135fdfd2ef8734baf7bcdabcbdccb1ab7: Status 404 returned error can't find the container with id 4fc80c9e4ebc41e5cd8f464f4d01f94135fdfd2ef8734baf7bcdabcbdccb1ab7
Oct 30 20:29:35.965771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:35.965722    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh" event={"ID":"5d876e30-bfc0-4db6-a136-e13e456d3abc","Type":"ContainerStarted","Data":"5eaf73045970d64a133a288e404ee6f24881457b86bcca4081bdff4e356d07ad"}
Oct 30 20:29:35.966143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:35.965801    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh" event={"ID":"5d876e30-bfc0-4db6-a136-e13e456d3abc","Type":"ContainerStarted","Data":"4fc80c9e4ebc41e5cd8f464f4d01f94135fdfd2ef8734baf7bcdabcbdccb1ab7"}
Oct 30 20:29:36.082267 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:36.082138    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh" podStartSLOduration=2.486871291 podStartE2EDuration="3.08211982s" totalImagesPullingTime="595.248529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:36.08097178 +0000 UTC m=+556.681773738" watchObservedRunningTime="2026-10-30 20:29:36.08211982 +0000 UTC m=+556.682921782"
Oct 30 20:29:36.581483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:36.581456    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:37.659635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.659591    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"]
Oct 30 20:29:37.663853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.663830    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675151    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ll4x\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675191    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675276    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675326    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675407    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675485 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675469    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675501    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675550 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675539    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.675583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.675569    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.715919 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.715882    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:29:37.716047 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.715938    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7k2d5\""
Oct 30 20:29:37.767130 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.767103    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"]
Oct 30 20:29:37.776034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776008    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.776203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776039    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.776203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776069    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.776203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776110    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ll4x\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.776203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776140    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.776203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776186    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776490    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776558    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776598    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.776836    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.777521    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.777582    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.777697    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.777800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.777727    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.779144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.779059    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.781458 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.781440    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.833481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.833446    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.833481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.833472    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ll4x\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cs6l\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:37.976644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:37.976593    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:38.427540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:38.427512    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"]
Oct 30 20:29:38.430203 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:29:38.430173    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode174b0ce_6120_43db_a840_bf69db117b7f.slice/crio-d856f4c1652701713c2d8c08e57ab6d3489b1c40f4e32dd669cbae81bf76e368 WatchSource:0}: Error finding container d856f4c1652701713c2d8c08e57ab6d3489b1c40f4e32dd669cbae81bf76e368: Status 404 returned error can't find the container with id d856f4c1652701713c2d8c08e57ab6d3489b1c40f4e32dd669cbae81bf76e368
Oct 30 20:29:38.993331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:38.993295    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" event={"ID":"e174b0ce-6120-43db-a840-bf69db117b7f","Type":"ContainerStarted","Data":"d856f4c1652701713c2d8c08e57ab6d3489b1c40f4e32dd669cbae81bf76e368"}
Oct 30 20:29:41.119630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:41.119595    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:41.133700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:41.133676    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:41.923242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:41.923210    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:29:41.923338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:41.923278    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:29:41.923338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:41.923304    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:29:42.011122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.011088    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" event={"ID":"e174b0ce-6120-43db-a840-bf69db117b7f","Type":"ContainerStarted","Data":"7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3"}
Oct 30 20:29:42.026735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.026668    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:29:42.148941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.148889    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podStartSLOduration=1.659239832 podStartE2EDuration="5.148874969s" totalImagesPullingTime="3.489635137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:42.147501052 +0000 UTC m=+562.748303013" watchObservedRunningTime="2026-10-30 20:29:42.148874969 +0000 UTC m=+562.749676922"
Oct 30 20:29:42.802086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.802042    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"]
Oct 30 20:29:42.805914 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.805898    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.827665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.827640    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.827796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.827694    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.827875 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.827856    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.827932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.827895    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qb4\" (UniqueName: \"kubernetes.io/projected/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-kube-api-access-m5qb4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.828004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.827988    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.828059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.828031    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.895765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.880976    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 20:29:42.911093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.911058    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"]
Oct 30 20:29:42.929132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929104    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qb4\" (UniqueName: \"kubernetes.io/projected/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-kube-api-access-m5qb4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929147    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929169    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929289    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929349    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929435    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929490    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929752 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929716    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.929840 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.929804    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.931456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.931430    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.931702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.931685    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:42.993766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.993122    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:42.995095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.995066    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:29:42.995208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.995121    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:29:42.996272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:42.996244    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qb4\" (UniqueName: \"kubernetes.io/projected/9bc25d7c-227f-4f78-9cfb-4cf5e3c69677-kube-api-access-m5qb4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jsdnb\" (UID: \"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:43.117964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:43.117877    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:43.515304 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:43.515273    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"]
Oct 30 20:29:43.518369 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:29:43.518338    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc25d7c_227f_4f78_9cfb_4cf5e3c69677.slice/crio-1eefeed9c5f5707b587f9ef0692b1b134d042b162825e9ad54966d939b659db3 WatchSource:0}: Error finding container 1eefeed9c5f5707b587f9ef0692b1b134d042b162825e9ad54966d939b659db3: Status 404 returned error can't find the container with id 1eefeed9c5f5707b587f9ef0692b1b134d042b162825e9ad54966d939b659db3
Oct 30 20:29:43.983773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:43.983145    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:29:43.983773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:43.983218    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:29:44.022887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:44.022848    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb" event={"ID":"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677","Type":"ContainerStarted","Data":"1eefeed9c5f5707b587f9ef0692b1b134d042b162825e9ad54966d939b659db3"}
Oct 30 20:29:44.593768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:44.592571    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:44.605784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:44.605757    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:44.988935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:44.988904    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:29:44.989199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:44.989172    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:29:45.029642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:45.029612    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb" event={"ID":"9bc25d7c-227f-4f78-9cfb-4cf5e3c69677","Type":"ContainerStarted","Data":"786e5b29d476821bf36af83e87665e1c737c8b0f5d94f48427b2675119a9a64e"}
Oct 30 20:29:45.040761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:45.040726    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh"
Oct 30 20:29:45.118849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:45.118818    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:45.149027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:45.148979    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb" podStartSLOduration=2.531779461 podStartE2EDuration="3.148965349s" totalImagesPullingTime="617.185888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:45.14752872 +0000 UTC m=+565.748330681" watchObservedRunningTime="2026-10-30 20:29:45.148965349 +0000 UTC m=+565.749767309"
Oct 30 20:29:45.981181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:45.981147    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:29:45.982136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:45.981729    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:29:46.978684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:46.978652    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:29:46.978990 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:46.978947    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:29:47.977320 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:47.977282    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:29:47.977787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:47.977334    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:29:49.408604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:49.408568    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:49.409170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:49.409146    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:49.409816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:49.409799    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 20:29:53.118306 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:53.118276    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:53.130218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:53.130194    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:54.082095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:54.082069    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jsdnb"
Oct 30 20:29:59.848434 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:59.848398    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"]
Oct 30 20:29:59.852750 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:59.852722    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:29:59.909405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:59.909366    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"]
Oct 30 20:29:59.956414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:59.956377    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 20:29:59.956560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:59.956513    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-stsf8\""
Oct 30 20:29:59.956560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:29:59.956519    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 20:30:00.014852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.014822    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl7cf\" (UniqueName: \"kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf\") pod \"ai-gateway-controller-5f6dd4f69f-stfj2\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.015004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.014955    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-stfj2\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.116201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.116121    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-stfj2\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.116346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.116231    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl7cf\" (UniqueName: \"kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf\") pod \"ai-gateway-controller-5f6dd4f69f-stfj2\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.118566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.118542    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-stfj2\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.161066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.161042    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"]
Oct 30 20:30:00.164308 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.164294    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.179837 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.179809    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl7cf\" (UniqueName: \"kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf\") pod \"ai-gateway-controller-5f6dd4f69f-stfj2\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.216255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.216234    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:00.216357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.216263    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:00.216357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.216295    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:30:00.221359 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.221335    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"]
Oct 30 20:30:00.317350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.317307    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-q99dr\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.317507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.317463    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtbrv\" (UniqueName: \"kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv\") pod \"maas-api-key-cleanup-29889870-q99dr\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.418415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.418327    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-q99dr\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.418415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.418395    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtbrv\" (UniqueName: \"kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv\") pod \"maas-api-key-cleanup-29889870-q99dr\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.418964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.418943    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-q99dr\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.465642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.465617    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:00.476019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.475995    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtbrv\" (UniqueName: \"kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv\") pod \"maas-api-key-cleanup-29889870-q99dr\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.482832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.482810    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:00.840379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.840354    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"]
Oct 30 20:30:00.842665 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:30:00.842638    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f541bc_a13c_4456_9df4_7bfbc1b8ec52.slice/crio-e880998ffb3e3908a5f65d9121c06f5bb4950cdd27c155657cacb17960b7a3a9 WatchSource:0}: Error finding container e880998ffb3e3908a5f65d9121c06f5bb4950cdd27c155657cacb17960b7a3a9: Status 404 returned error can't find the container with id e880998ffb3e3908a5f65d9121c06f5bb4950cdd27c155657cacb17960b7a3a9
Oct 30 20:30:00.889039 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:00.888986    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"]
Oct 30 20:30:00.891197 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:30:00.891171    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405a339a_071e_4f1e_9702_ded254851360.slice/crio-6b9971d17c82e4d766ab9bd4e218b84a3275d9a618596a18b495e7bfeab62cdb WatchSource:0}: Error finding container 6b9971d17c82e4d766ab9bd4e218b84a3275d9a618596a18b495e7bfeab62cdb: Status 404 returned error can't find the container with id 6b9971d17c82e4d766ab9bd4e218b84a3275d9a618596a18b495e7bfeab62cdb
Oct 30 20:30:01.113375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:01.113292    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2" event={"ID":"11f541bc-a13c-4456-9df4-7bfbc1b8ec52","Type":"ContainerStarted","Data":"e880998ffb3e3908a5f65d9121c06f5bb4950cdd27c155657cacb17960b7a3a9"}
Oct 30 20:30:01.114618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:01.114591    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerStarted","Data":"6b9971d17c82e4d766ab9bd4e218b84a3275d9a618596a18b495e7bfeab62cdb"}
Oct 30 20:30:01.979031 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:01.978999    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:30:03.248246 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.248215    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-745f99f494-cphzg"]
Oct 30 20:30:03.248673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.248455    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-745f99f494-cphzg" podUID="d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" containerName="authorino" containerID="cri-o://81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177" gracePeriod=30
Oct 30 20:30:03.534002 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.533976    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:30:03.651498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.651464    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8\" (UniqueName: \"kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8\") pod \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") "
Oct 30 20:30:03.651683 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.651529    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert\" (UniqueName: \"kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert\") pod \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\" (UID: \"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18\") "
Oct 30 20:30:03.655352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.655322    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8" pod "d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" (UID: "d5f7ca12-d2d6-4a41-8396-1f635f9d8f18"). InnerVolumeSpecName "kube-api-access-zqkf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:03.667128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.667093    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert" pod "d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" (UID: "d5f7ca12-d2d6-4a41-8396-1f635f9d8f18"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:03.752811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.752776    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqkf8\" (UniqueName: \"kubernetes.io/projected/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-kube-api-access-zqkf8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:03.752811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:03.752812    5967 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18-tls-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:04.132169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.132133    5967 generic.go:356] "Generic (PLEG): container finished" podID="d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" containerID="81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177" exitCode=0
Oct 30 20:30:04.132326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.132176    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-745f99f494-cphzg"
Oct 30 20:30:04.132326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.132213    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-745f99f494-cphzg" event={"ID":"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18","Type":"ContainerDied","Data":"81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177"}
Oct 30 20:30:04.132326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.132258    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-745f99f494-cphzg" event={"ID":"d5f7ca12-d2d6-4a41-8396-1f635f9d8f18","Type":"ContainerDied","Data":"b98c375d28c04aebf924b63dd9691dce99d63ce6e466bc033ffed8fabfb30dec"}
Oct 30 20:30:04.132326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.132275    5967 scope.go:122] "RemoveContainer" containerID="81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177"
Oct 30 20:30:04.142626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.142337    5967 scope.go:122] "RemoveContainer" containerID="81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177"
Oct 30 20:30:04.142698 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:04.142664    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177\": container with ID starting with 81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177 not found: ID does not exist" containerID="81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177"
Oct 30 20:30:04.142772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.142688    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177"} err="failed to get container status \"81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177\": rpc error: code = NotFound desc = could not find container \"81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177\": container with ID starting with 81f369da33185c47a117f36347fedd615d537a73af2c8354db1e06bf273aa177 not found: ID does not exist"
Oct 30 20:30:04.298626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.298597    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-745f99f494-cphzg"]
Oct 30 20:30:04.347259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:04.347232    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-745f99f494-cphzg"]
Oct 30 20:30:05.544229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:05.544194    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" path="/var/lib/kubelet/pods/d5f7ca12-d2d6-4a41-8396-1f635f9d8f18/volumes"
Oct 30 20:30:07.154285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:07.154250    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerStarted","Data":"6699366d8780288ad9ef6b22fa1cfa1d5b200b04761a2db2fe84a506d3cdbc0a"}
Oct 30 20:30:07.155642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:07.155620    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2" event={"ID":"11f541bc-a13c-4456-9df4-7bfbc1b8ec52","Type":"ContainerStarted","Data":"7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b"}
Oct 30 20:30:07.155792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:07.155775    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:07.269158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:07.269109    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" podStartSLOduration=1.338336298 podStartE2EDuration="7.269096028s" totalImagesPullingTime="5.93075973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:07.267121784 +0000 UTC m=+587.867923745" watchObservedRunningTime="2026-10-30 20:30:07.269096028 +0000 UTC m=+587.869897987"
Oct 30 20:30:07.384264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:07.384181    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2" podStartSLOduration=2.692328622 podStartE2EDuration="8.384168182s" totalImagesPullingTime="5.69183956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:07.382930506 +0000 UTC m=+587.983732466" watchObservedRunningTime="2026-10-30 20:30:07.384168182 +0000 UTC m=+587.984970141"
Oct 30 20:30:08.163085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:08.163053    5967 generic.go:356] "Generic (PLEG): container finished" podID="405a339a-071e-4f1e-9702-ded254851360" containerID="6699366d8780288ad9ef6b22fa1cfa1d5b200b04761a2db2fe84a506d3cdbc0a" exitCode=60
Oct 30 20:30:08.163501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:08.163143    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerDied","Data":"6699366d8780288ad9ef6b22fa1cfa1d5b200b04761a2db2fe84a506d3cdbc0a"}
Oct 30 20:30:08.163501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:08.163326    5967 scope.go:122] "RemoveContainer" containerID="6699366d8780288ad9ef6b22fa1cfa1d5b200b04761a2db2fe84a506d3cdbc0a"
Oct 30 20:30:09.176182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:09.176151    5967 generic.go:356] "Generic (PLEG): container finished" podID="405a339a-071e-4f1e-9702-ded254851360" containerID="b710e325aa3de858099af1f4297906316689f31ed7aa206d1d5b37b78a982878" exitCode=60
Oct 30 20:30:09.176556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:09.176238    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerDied","Data":"b710e325aa3de858099af1f4297906316689f31ed7aa206d1d5b37b78a982878"}
Oct 30 20:30:09.176556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:09.176308    5967 scope.go:122] "RemoveContainer" containerID="6699366d8780288ad9ef6b22fa1cfa1d5b200b04761a2db2fe84a506d3cdbc0a"
Oct 30 20:30:09.176556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:09.176507    5967 scope.go:122] "RemoveContainer" containerID="b710e325aa3de858099af1f4297906316689f31ed7aa206d1d5b37b78a982878"
Oct 30 20:30:09.176806 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:09.176775    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-q99dr_odh-ai-gateway-infra(405a339a-071e-4f1e-9702-ded254851360)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" podUID="405a339a-071e-4f1e-9702-ded254851360"
Oct 30 20:30:10.183091 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.183060    5967 scope.go:122] "RemoveContainer" containerID="b710e325aa3de858099af1f4297906316689f31ed7aa206d1d5b37b78a982878"
Oct 30 20:30:10.183467 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:10.183265    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-q99dr_odh-ai-gateway-infra(405a339a-071e-4f1e-9702-ded254851360)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" podUID="405a339a-071e-4f1e-9702-ded254851360"
Oct 30 20:30:10.903120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.903085    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"]
Oct 30 20:30:10.903576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.903562    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" containerName="authorino"
Oct 30 20:30:10.903620 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.903579    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" containerName="authorino"
Oct 30 20:30:10.903653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.903646    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f7ca12-d2d6-4a41-8396-1f635f9d8f18" containerName="authorino"
Oct 30 20:30:10.906240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.906221    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:10.921233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.921206    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da87480-60d8-45a3-aeda-895907c9e229-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9bl76\" (UID: \"4da87480-60d8-45a3-aeda-895907c9e229\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:10.921355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.921253    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcxn\" (UniqueName: \"kubernetes.io/projected/4da87480-60d8-45a3-aeda-895907c9e229-kube-api-access-qgcxn\") pod \"payload-pre-processing-7b98bb678c-9bl76\" (UID: \"4da87480-60d8-45a3-aeda-895907c9e229\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:10.957871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.957846    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 20:30:10.957871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:10.957862    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zjprb\""
Oct 30 20:30:11.008583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.008550    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"]
Oct 30 20:30:11.022724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.022690    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da87480-60d8-45a3-aeda-895907c9e229-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9bl76\" (UID: \"4da87480-60d8-45a3-aeda-895907c9e229\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:11.022908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.022757    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcxn\" (UniqueName: \"kubernetes.io/projected/4da87480-60d8-45a3-aeda-895907c9e229-kube-api-access-qgcxn\") pod \"payload-pre-processing-7b98bb678c-9bl76\" (UID: \"4da87480-60d8-45a3-aeda-895907c9e229\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:11.023392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.023374    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da87480-60d8-45a3-aeda-895907c9e229-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9bl76\" (UID: \"4da87480-60d8-45a3-aeda-895907c9e229\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:11.063063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.063031    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-vtz7b"]
Oct 30 20:30:11.066006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.065985    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.081704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.081679    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcxn\" (UniqueName: \"kubernetes.io/projected/4da87480-60d8-45a3-aeda-895907c9e229-kube-api-access-qgcxn\") pod \"payload-pre-processing-7b98bb678c-9bl76\" (UID: \"4da87480-60d8-45a3-aeda-895907c9e229\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:11.123822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.123798    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-vtz7b"]
Oct 30 20:30:11.124077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.124054    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20b8d66-af6a-4db2-a44a-f20e16cb834e-plugins-config-volume\") pod \"payload-processing-5f78687d5b-vtz7b\" (UID: \"b20b8d66-af6a-4db2-a44a-f20e16cb834e\") " pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.124197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.124098    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679d7\" (UniqueName: \"kubernetes.io/projected/b20b8d66-af6a-4db2-a44a-f20e16cb834e-kube-api-access-679d7\") pod \"payload-processing-5f78687d5b-vtz7b\" (UID: \"b20b8d66-af6a-4db2-a44a-f20e16cb834e\") " pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.217829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.217807    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:11.225131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.225109    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20b8d66-af6a-4db2-a44a-f20e16cb834e-plugins-config-volume\") pod \"payload-processing-5f78687d5b-vtz7b\" (UID: \"b20b8d66-af6a-4db2-a44a-f20e16cb834e\") " pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.225237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.225155    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679d7\" (UniqueName: \"kubernetes.io/projected/b20b8d66-af6a-4db2-a44a-f20e16cb834e-kube-api-access-679d7\") pod \"payload-processing-5f78687d5b-vtz7b\" (UID: \"b20b8d66-af6a-4db2-a44a-f20e16cb834e\") " pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.225865 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.225847    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20b8d66-af6a-4db2-a44a-f20e16cb834e-plugins-config-volume\") pod \"payload-processing-5f78687d5b-vtz7b\" (UID: \"b20b8d66-af6a-4db2-a44a-f20e16cb834e\") " pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.282347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.281869    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-679d7\" (UniqueName: \"kubernetes.io/projected/b20b8d66-af6a-4db2-a44a-f20e16cb834e-kube-api-access-679d7\") pod \"payload-processing-5f78687d5b-vtz7b\" (UID: \"b20b8d66-af6a-4db2-a44a-f20e16cb834e\") " pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.378315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.378284    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:11.586485 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.586463    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"]
Oct 30 20:30:11.588952 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:30:11.588929    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da87480_60d8_45a3_aeda_895907c9e229.slice/crio-4732edfeaa00a39fb5f0933fc170b1971982bd91358db92948d5f8acd00a697c WatchSource:0}: Error finding container 4732edfeaa00a39fb5f0933fc170b1971982bd91358db92948d5f8acd00a697c: Status 404 returned error can't find the container with id 4732edfeaa00a39fb5f0933fc170b1971982bd91358db92948d5f8acd00a697c
Oct 30 20:30:11.748620 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:11.748602    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-vtz7b"]
Oct 30 20:30:11.751109 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:30:11.751081    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20b8d66_af6a_4db2_a44a_f20e16cb834e.slice/crio-25a99bf77903f8a1d250c1cb8216cb0787be7303c25e64f9931bd2d741b25c85 WatchSource:0}: Error finding container 25a99bf77903f8a1d250c1cb8216cb0787be7303c25e64f9931bd2d741b25c85: Status 404 returned error can't find the container with id 25a99bf77903f8a1d250c1cb8216cb0787be7303c25e64f9931bd2d741b25c85
Oct 30 20:30:12.193152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:12.193123    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76" event={"ID":"4da87480-60d8-45a3-aeda-895907c9e229","Type":"ContainerStarted","Data":"4732edfeaa00a39fb5f0933fc170b1971982bd91358db92948d5f8acd00a697c"}
Oct 30 20:30:12.194325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:12.194297    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b" event={"ID":"b20b8d66-af6a-4db2-a44a-f20e16cb834e","Type":"ContainerStarted","Data":"25a99bf77903f8a1d250c1cb8216cb0787be7303c25e64f9931bd2d741b25c85"}
Oct 30 20:30:15.213956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:15.213920    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b" event={"ID":"b20b8d66-af6a-4db2-a44a-f20e16cb834e","Type":"ContainerStarted","Data":"4dd5f83b94094f8eb139fc2cc6cc5b192e6d498837c487c5c247b2a7bd9e804c"}
Oct 30 20:30:15.214358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:15.214025    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:15.215439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:15.215406    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76" event={"ID":"4da87480-60d8-45a3-aeda-895907c9e229","Type":"ContainerStarted","Data":"094e109047b7cf171f1c528c9bb65aa1f7d4c56150d90902990f5e5ff47b56e8"}
Oct 30 20:30:15.215552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:15.215537    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:15.329360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:15.329203    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b" podStartSLOduration=1.632918177 podStartE2EDuration="4.329186006s" totalImagesPullingTime="2.696267829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:15.326814572 +0000 UTC m=+595.927616533" watchObservedRunningTime="2026-10-30 20:30:15.329186006 +0000 UTC m=+595.929987966"
Oct 30 20:30:15.441151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:15.441106    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76" podStartSLOduration=2.666164618 podStartE2EDuration="5.441092051s" totalImagesPullingTime="2.774927433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:15.4395114 +0000 UTC m=+596.040313362" watchObservedRunningTime="2026-10-30 20:30:15.441092051 +0000 UTC m=+596.041894008"
Oct 30 20:30:18.165771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.165729    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:18.519355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.519321    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-85b76df47b-hf9k5"]
Oct 30 20:30:18.522253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.522231    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.579976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.579956    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-85b76df47b-hf9k5"]
Oct 30 20:30:18.597637 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.597613    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b81ab9-ce89-45f4-8a12-37447542b8a4-metrics-certs\") pod \"ai-gateway-controller-85b76df47b-hf9k5\" (UID: \"08b81ab9-ce89-45f4-8a12-37447542b8a4\") " pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.597901 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.597880    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4l5s\" (UniqueName: \"kubernetes.io/projected/08b81ab9-ce89-45f4-8a12-37447542b8a4-kube-api-access-v4l5s\") pod \"ai-gateway-controller-85b76df47b-hf9k5\" (UID: \"08b81ab9-ce89-45f4-8a12-37447542b8a4\") " pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.698589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.698565    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b81ab9-ce89-45f4-8a12-37447542b8a4-metrics-certs\") pod \"ai-gateway-controller-85b76df47b-hf9k5\" (UID: \"08b81ab9-ce89-45f4-8a12-37447542b8a4\") " pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.698695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.698606    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4l5s\" (UniqueName: \"kubernetes.io/projected/08b81ab9-ce89-45f4-8a12-37447542b8a4-kube-api-access-v4l5s\") pod \"ai-gateway-controller-85b76df47b-hf9k5\" (UID: \"08b81ab9-ce89-45f4-8a12-37447542b8a4\") " pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.700994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.700976    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b81ab9-ce89-45f4-8a12-37447542b8a4-metrics-certs\") pod \"ai-gateway-controller-85b76df47b-hf9k5\" (UID: \"08b81ab9-ce89-45f4-8a12-37447542b8a4\") " pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.754560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.754540    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4l5s\" (UniqueName: \"kubernetes.io/projected/08b81ab9-ce89-45f4-8a12-37447542b8a4-kube-api-access-v4l5s\") pod \"ai-gateway-controller-85b76df47b-hf9k5\" (UID: \"08b81ab9-ce89-45f4-8a12-37447542b8a4\") " pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:18.834100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:18.834039    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:19.201495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:19.201395    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-85b76df47b-hf9k5"]
Oct 30 20:30:19.204227 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:30:19.204198    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b81ab9_ce89_45f4_8a12_37447542b8a4.slice/crio-7e0729f79a9f4b54f00af3c65f89b3346de3b47e3fa55d338123662a496a624e WatchSource:0}: Error finding container 7e0729f79a9f4b54f00af3c65f89b3346de3b47e3fa55d338123662a496a624e: Status 404 returned error can't find the container with id 7e0729f79a9f4b54f00af3c65f89b3346de3b47e3fa55d338123662a496a624e
Oct 30 20:30:19.235085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:19.235052    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5" event={"ID":"08b81ab9-ce89-45f4-8a12-37447542b8a4","Type":"ContainerStarted","Data":"7e0729f79a9f4b54f00af3c65f89b3346de3b47e3fa55d338123662a496a624e"}
Oct 30 20:30:20.241878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:20.241835    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5" event={"ID":"08b81ab9-ce89-45f4-8a12-37447542b8a4","Type":"ContainerStarted","Data":"dd60362ce5d941372ac3cf6f0a7c370963beac5dbd737df3f5dcb302957c5c45"}
Oct 30 20:30:20.242343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:20.241971    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:20.358407 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:20.358316    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5" podStartSLOduration=1.539510011 podStartE2EDuration="2.358302887s" totalImagesPullingTime="818.792876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:20.356248928 +0000 UTC m=+600.957050889" watchObservedRunningTime="2026-10-30 20:30:20.358302887 +0000 UTC m=+600.959104880"
Oct 30 20:30:22.535324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:22.535295    5967 scope.go:122] "RemoveContainer" containerID="b710e325aa3de858099af1f4297906316689f31ed7aa206d1d5b37b78a982878"
Oct 30 20:30:23.257832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:23.257793    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerStarted","Data":"6762a57300a64d23ba114eb207ddeda398e22292a137d854e841c6773afe6257"}
Oct 30 20:30:24.264758 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:24.264719    5967 generic.go:356] "Generic (PLEG): container finished" podID="405a339a-071e-4f1e-9702-ded254851360" containerID="6762a57300a64d23ba114eb207ddeda398e22292a137d854e841c6773afe6257" exitCode=60
Oct 30 20:30:24.265212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:24.264782    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerDied","Data":"6762a57300a64d23ba114eb207ddeda398e22292a137d854e841c6773afe6257"}
Oct 30 20:30:24.265212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:24.264843    5967 scope.go:122] "RemoveContainer" containerID="b710e325aa3de858099af1f4297906316689f31ed7aa206d1d5b37b78a982878"
Oct 30 20:30:24.265212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:24.265200    5967 scope.go:122] "RemoveContainer" containerID="6762a57300a64d23ba114eb207ddeda398e22292a137d854e841c6773afe6257"
Oct 30 20:30:24.265444 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:24.265425    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-q99dr_odh-ai-gateway-infra(405a339a-071e-4f1e-9702-ded254851360)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" podUID="405a339a-071e-4f1e-9702-ded254851360"
Oct 30 20:30:24.379827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:24.379803    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"]
Oct 30 20:30:25.407184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.407160    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:25.457863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.457838    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv\" (UniqueName: \"kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv\") pod \"405a339a-071e-4f1e-9702-ded254851360\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") "
Oct 30 20:30:25.457983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.457879    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca\") pod \"405a339a-071e-4f1e-9702-ded254851360\" (UID: \"405a339a-071e-4f1e-9702-ded254851360\") "
Oct 30 20:30:25.458251 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.458203    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca" pod "405a339a-071e-4f1e-9702-ded254851360" (UID: "405a339a-071e-4f1e-9702-ded254851360"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:25.460019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.459999    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv" pod "405a339a-071e-4f1e-9702-ded254851360" (UID: "405a339a-071e-4f1e-9702-ded254851360"). InnerVolumeSpecName "kube-api-access-qtbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:25.559419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.559394    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtbrv\" (UniqueName: \"kubernetes.io/projected/405a339a-071e-4f1e-9702-ded254851360-kube-api-access-qtbrv\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:25.559419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:25.559417    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405a339a-071e-4f1e-9702-ded254851360-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:26.222789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.222761    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-vtz7b"
Oct 30 20:30:26.222957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.222836    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9bl76"
Oct 30 20:30:26.276558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.276533    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"
Oct 30 20:30:26.276558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.276551    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr" event={"ID":"405a339a-071e-4f1e-9702-ded254851360","Type":"ContainerDied","Data":"6b9971d17c82e4d766ab9bd4e218b84a3275d9a618596a18b495e7bfeab62cdb"}
Oct 30 20:30:26.276735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.276608    5967 scope.go:122] "RemoveContainer" containerID="6762a57300a64d23ba114eb207ddeda398e22292a137d854e841c6773afe6257"
Oct 30 20:30:26.446492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.446462    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"]
Oct 30 20:30:26.450209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:26.450184    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q99dr"]
Oct 30 20:30:27.540641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:27.540598    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405a339a-071e-4f1e-9702-ded254851360" path="/var/lib/kubelet/pods/405a339a-071e-4f1e-9702-ded254851360/volumes"
Oct 30 20:30:31.248148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.248122    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-85b76df47b-hf9k5"
Oct 30 20:30:31.413868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.413837    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"]
Oct 30 20:30:31.414037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.414018    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2" podUID="11f541bc-a13c-4456-9df4-7bfbc1b8ec52" containerName="manager" containerID="cri-o://7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b" gracePeriod=30
Oct 30 20:30:31.663606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.663582    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:31.815687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.815607    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf\" (UniqueName: \"kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf\") pod \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") "
Oct 30 20:30:31.815687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.815656    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs\") pod \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\" (UID: \"11f541bc-a13c-4456-9df4-7bfbc1b8ec52\") "
Oct 30 20:30:31.817951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.817917    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs" pod "11f541bc-a13c-4456-9df4-7bfbc1b8ec52" (UID: "11f541bc-a13c-4456-9df4-7bfbc1b8ec52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:31.818059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.818007    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf" pod "11f541bc-a13c-4456-9df4-7bfbc1b8ec52" (UID: "11f541bc-a13c-4456-9df4-7bfbc1b8ec52"). InnerVolumeSpecName "kube-api-access-hl7cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:31.916677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.916650    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl7cf\" (UniqueName: \"kubernetes.io/projected/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-kube-api-access-hl7cf\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:31.916677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:31.916675    5967 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f541bc-a13c-4456-9df4-7bfbc1b8ec52-metrics-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:31.979210 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:31.979177    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:30:32.305887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.305858    5967 generic.go:356] "Generic (PLEG): container finished" podID="11f541bc-a13c-4456-9df4-7bfbc1b8ec52" containerID="7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b" exitCode=0
Oct 30 20:30:32.306317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.305914    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"
Oct 30 20:30:32.306317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.305937    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2" event={"ID":"11f541bc-a13c-4456-9df4-7bfbc1b8ec52","Type":"ContainerDied","Data":"7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b"}
Oct 30 20:30:32.306317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.306034    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2" event={"ID":"11f541bc-a13c-4456-9df4-7bfbc1b8ec52","Type":"ContainerDied","Data":"e880998ffb3e3908a5f65d9121c06f5bb4950cdd27c155657cacb17960b7a3a9"}
Oct 30 20:30:32.306317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.306067    5967 scope.go:122] "RemoveContainer" containerID="7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b"
Oct 30 20:30:32.315125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.315108    5967 scope.go:122] "RemoveContainer" containerID="7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b"
Oct 30 20:30:32.315448 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:30:32.315431    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b\": container with ID starting with 7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b not found: ID does not exist" containerID="7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b"
Oct 30 20:30:32.315484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.315457    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b"} err="failed to get container status \"7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b\": rpc error: code = NotFound desc = could not find container \"7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b\": container with ID starting with 7cdc21583fb9907f510da6c6a6c88be8206b73225499014e9fad758db346d52b not found: ID does not exist"
Oct 30 20:30:32.467884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.467861    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"]
Oct 30 20:30:32.516933 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:32.516909    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-stfj2"]
Oct 30 20:30:33.541085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:30:33.541053    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f541bc-a13c-4456-9df4-7bfbc1b8ec52" path="/var/lib/kubelet/pods/11f541bc-a13c-4456-9df4-7bfbc1b8ec52/volumes"
Oct 30 20:31:01.978723 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:31:01.978647    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:31:04.240295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240263    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"]
Oct 30 20:31:04.240696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240683    5967 cpu_manager.go:414] "containerMap: removing container" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240696    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240706    5967 cpu_manager.go:414] "containerMap: removing container" podUID="11f541bc-a13c-4456-9df4-7bfbc1b8ec52" containerName="manager"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240711    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f541bc-a13c-4456-9df4-7bfbc1b8ec52" containerName="manager"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240727    5967 cpu_manager.go:414] "containerMap: removing container" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240732    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240756    5967 cpu_manager.go:414] "containerMap: removing container" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.240775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240761    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.241020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240837    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.241020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240849    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.241020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.240855    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f541bc-a13c-4456-9df4-7bfbc1b8ec52" containerName="manager"
Oct 30 20:31:04.243898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.243883    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.295667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.295643    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-df031b-openshift-default-dockercfg-8m6p4\""
Oct 30 20:31:04.302104 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302074    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302126    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302271    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7g27\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302304    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302337    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302440 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302420    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302466    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302689    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.302809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.302794    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.346527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.346505    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"]
Oct 30 20:31:04.404127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404100    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404141    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404183    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404238    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7g27\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404270    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404292    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404339    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404378    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404404    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404531    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404756    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404901    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.404994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.404935    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.405079 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.405039    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.406804 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.406786    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.407018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.406998    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.461482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.461456    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.461614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.461577    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7g27\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.556342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.556277    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:04.899302 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.899232    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"]
Oct 30 20:31:04.899756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.899725    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="405a339a-071e-4f1e-9702-ded254851360" containerName="cleanup"
Oct 30 20:31:04.903712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.903695    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:04.957095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.957071    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0a78ac-openshift-default-dockercfg-6pwsb\""
Oct 30 20:31:04.965526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.965500    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"]
Oct 30 20:31:04.988035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.988009    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"]
Oct 30 20:31:04.992039 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:04.992017    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8324624e_fbd0_4fc1_92c1_05c60d635176.slice/crio-3d5f762b3c3208c5a2e12c690ac9f86fd7c25220d5024f388dd03b5f22bdc13b WatchSource:0}: Error finding container 3d5f762b3c3208c5a2e12c690ac9f86fd7c25220d5024f388dd03b5f22bdc13b: Status 404 returned error can't find the container with id 3d5f762b3c3208c5a2e12c690ac9f86fd7c25220d5024f388dd03b5f22bdc13b
Oct 30 20:31:04.994652 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.994618    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:04.994733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.994685    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:04.994733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:04.994714    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.009033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009010    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009074    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009164    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009198    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009239    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009333    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009371    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009479    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.009544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.009524    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77r8x\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.110996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.110959    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111022    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111058    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111087    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111125    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111144    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111166    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111183    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111204    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77r8x\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111370    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.111772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111731    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.112014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.111995    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.112082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.112005    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.112189 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.112174    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.113761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.113722    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.113906 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.113784    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.171250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.171188    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.171381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.171328    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77r8x\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.216502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.216475    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:05.468391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.468356    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" event={"ID":"8324624e-fbd0-4fc1-92c1-05c60d635176","Type":"ContainerStarted","Data":"f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"}
Oct 30 20:31:05.468862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.468402    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" event={"ID":"8324624e-fbd0-4fc1-92c1-05c60d635176","Type":"ContainerStarted","Data":"3d5f762b3c3208c5a2e12c690ac9f86fd7c25220d5024f388dd03b5f22bdc13b"}
Oct 30 20:31:05.556383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.556361    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:05.614912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.614864    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podStartSLOduration=1.614851642 podStartE2EDuration="1.614851642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:05.612417866 +0000 UTC m=+646.213219826" watchObservedRunningTime="2026-10-30 20:31:05.614851642 +0000 UTC m=+646.215653602"
Oct 30 20:31:05.659108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.659078    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"]
Oct 30 20:31:05.664244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.664221    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"]
Oct 30 20:31:05.664374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.664343    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.668695 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:05.668675    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb7b62c_26d5_4657_b164_c1549cb4942d.slice/crio-f5e93cdc8416d927858df7fc59f81e25d69ae33ca372bf6eb4ab9a8aa93e9158 WatchSource:0}: Error finding container f5e93cdc8416d927858df7fc59f81e25d69ae33ca372bf6eb4ab9a8aa93e9158: Status 404 returned error can't find the container with id f5e93cdc8416d927858df7fc59f81e25d69ae33ca372bf6eb4ab9a8aa93e9158
Oct 30 20:31:05.671426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.671393    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.671554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.671466    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.671554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.671507    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:05.716392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716371    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716499 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716399    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716499 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716428    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716499 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716444    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjmq\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716570    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716624    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716729    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716872    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.716930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.716900    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.717055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.717037    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8e2fc1-openshift-default-dockercfg-77c66\""
Oct 30 20:31:05.727390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.727367    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"]
Oct 30 20:31:05.817642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.817608    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.817826 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.817664    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.817826 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.817792    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.817826 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.817818    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.817995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.817974    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818064    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818105    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818204    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818230    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjmq\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818275    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818291    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818424    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.818524 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.818480    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.819200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.819177    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.820548 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.820519    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.820655 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.820632    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.874552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.874525    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:05.874931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:05.874912    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjmq\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:06.016036 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.015960    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:06.306635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.306566    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"]
Oct 30 20:31:06.312155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.312135    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.376975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.376941    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"]
Oct 30 20:31:06.422731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.422711    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"]
Oct 30 20:31:06.425121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425086    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425165    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425523 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:06.425493    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc9a219_f3e6_4bd8_8cb8_0856316cf0d7.slice/crio-07bee2be673316674128476d9c1d8800fb6f8fb6a9a2ebef3b08a614489005d5 WatchSource:0}: Error finding container 07bee2be673316674128476d9c1d8800fb6f8fb6a9a2ebef3b08a614489005d5: Status 404 returned error can't find the container with id 07bee2be673316674128476d9c1d8800fb6f8fb6a9a2ebef3b08a614489005d5
Oct 30 20:31:06.425681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425660    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425701    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425770    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425885    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425907    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hc5l\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.425940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425926    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.426090 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.425941    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.427934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.427902    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.428028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.427963    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.428028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.427993    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:06.444858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.444841    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a78bf4-openshift-default-dockercfg-cn9sv\""
Oct 30 20:31:06.476398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.476361    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" event={"ID":"4fb7b62c-26d5-4657-b164-c1549cb4942d","Type":"ContainerStarted","Data":"bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"}
Oct 30 20:31:06.476787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.476425    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" event={"ID":"4fb7b62c-26d5-4657-b164-c1549cb4942d","Type":"ContainerStarted","Data":"f5e93cdc8416d927858df7fc59f81e25d69ae33ca372bf6eb4ab9a8aa93e9158"}
Oct 30 20:31:06.478614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.478221    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" event={"ID":"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7","Type":"ContainerStarted","Data":"07bee2be673316674128476d9c1d8800fb6f8fb6a9a2ebef3b08a614489005d5"}
Oct 30 20:31:06.527329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527300    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527339    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hc5l\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527369    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527392    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527431    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527596    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527668    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527783    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.527884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.527848    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.528108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.528015    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.528449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.528422    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.528536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.528501    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.529140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.528916    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.529140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.529100    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.529928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.529907    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.531186 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.531167    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.562779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.562704    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:06.584475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.584441    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hc5l\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.585106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.585089    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:06.607964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.607923    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podStartSLOduration=2.60791155 podStartE2EDuration="2.60791155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:06.605928093 +0000 UTC m=+647.206730056" watchObservedRunningTime="2026-10-30 20:31:06.60791155 +0000 UTC m=+647.208713509"
Oct 30 20:31:06.625375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:06.625351    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:07.009894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.009614    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"]
Oct 30 20:31:07.012284 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:07.012257    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f0e4ca_0fde_4823_a54d_72663bbf30ed.slice/crio-36bf04ed8ed866199cac3161465cdbba8b5d03b0073a3da3a503b8e6b812ce67 WatchSource:0}: Error finding container 36bf04ed8ed866199cac3161465cdbba8b5d03b0073a3da3a503b8e6b812ce67: Status 404 returned error can't find the container with id 36bf04ed8ed866199cac3161465cdbba8b5d03b0073a3da3a503b8e6b812ce67
Oct 30 20:31:07.014518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.014482    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:07.014625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.014561    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:07.014625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.014605    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:07.216928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.216894    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:07.221796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.221772    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:07.484347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.484309    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" event={"ID":"48f0e4ca-0fde-4823-a54d-72663bbf30ed","Type":"ContainerStarted","Data":"26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"}
Oct 30 20:31:07.484813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.484365    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" event={"ID":"48f0e4ca-0fde-4823-a54d-72663bbf30ed","Type":"ContainerStarted","Data":"36bf04ed8ed866199cac3161465cdbba8b5d03b0073a3da3a503b8e6b812ce67"}
Oct 30 20:31:07.485889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.485864    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" event={"ID":"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7","Type":"ContainerStarted","Data":"ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"}
Oct 30 20:31:07.486103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.486076    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:07.486290 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.486257    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:07.487295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.487274    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:07.487388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.487334    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:07.606477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.606425    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podStartSLOduration=1.6064083249999999 podStartE2EDuration="1.606408325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:07.604888587 +0000 UTC m=+648.205690557" watchObservedRunningTime="2026-10-30 20:31:07.606408325 +0000 UTC m=+648.207210285"
Oct 30 20:31:07.625777 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.625755    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:07.735474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:07.735384    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podStartSLOduration=2.7353720040000002 podStartE2EDuration="2.735372004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:07.730078531 +0000 UTC m=+648.330880491" watchObservedRunningTime="2026-10-30 20:31:07.735372004 +0000 UTC m=+648.336173963"
Oct 30 20:31:08.016732 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:08.016653    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:08.022726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:08.022703    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:08.492073 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:08.492049    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:08.492940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:08.492919    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:31:08.631722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:08.631692    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:09.496891 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:09.496866    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:09.498023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:09.498002    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:31:13.297437 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.297407    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"]
Oct 30 20:31:13.301436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.301417    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.353076 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.353056    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-df031b\""
Oct 30 20:31:13.400523 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.400480    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xkfv\" (UniqueName: \"kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv\") pod \"payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.400962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.400938    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.405856 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.405814    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"]
Oct 30 20:31:13.451735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.451705    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-df031b-dockercfg-89fww\""
Oct 30 20:31:13.464152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.464129    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"]
Oct 30 20:31:13.467701 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.467684    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.501993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.501970    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xkfv\" (UniqueName: \"kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv\") pod \"payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.502084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.502048    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.502676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.502656    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.527735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.527712    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"]
Oct 30 20:31:13.560795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.560698    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xkfv\" (UniqueName: \"kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv\") pod \"payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.602835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.602804    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzxt\" (UniqueName: \"kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt\") pod \"payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.602940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.602887    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.614692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.614665    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:13.704496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.704451    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzxt\" (UniqueName: \"kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt\") pod \"payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.704672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.704538    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.705420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.705391    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.771399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.771369    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzxt\" (UniqueName: \"kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt\") pod \"payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:13.781704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:13.781683    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:14.002848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.002823    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"]
Oct 30 20:31:14.004357 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:14.004324    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84156bc_3e2c_46c4_9c5f_ad5d2e91342d.slice/crio-0ec1e503070c8c50794a073aa122437d1e4796c70f99cc06a6fcda53a5bc9dc0 WatchSource:0}: Error finding container 0ec1e503070c8c50794a073aa122437d1e4796c70f99cc06a6fcda53a5bc9dc0: Status 404 returned error can't find the container with id 0ec1e503070c8c50794a073aa122437d1e4796c70f99cc06a6fcda53a5bc9dc0
Oct 30 20:31:14.161096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.161027    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"]
Oct 30 20:31:14.164330 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:14.164306    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954ad950_625d_4ae9_b98f_9b8c54017993.slice/crio-14eb429d161a3a87056b0dbd863032e355f34ab35622f7ff822cb53ebbb6d648 WatchSource:0}: Error finding container 14eb429d161a3a87056b0dbd863032e355f34ab35622f7ff822cb53ebbb6d648: Status 404 returned error can't find the container with id 14eb429d161a3a87056b0dbd863032e355f34ab35622f7ff822cb53ebbb6d648
Oct 30 20:31:14.523406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.523376    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" event={"ID":"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d","Type":"ContainerStarted","Data":"ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e"}
Oct 30 20:31:14.523864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.523428    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" event={"ID":"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d","Type":"ContainerStarted","Data":"0ec1e503070c8c50794a073aa122437d1e4796c70f99cc06a6fcda53a5bc9dc0"}
Oct 30 20:31:14.523864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.523539    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:14.525038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.525014    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" event={"ID":"954ad950-625d-4ae9-b98f-9b8c54017993","Type":"ContainerStarted","Data":"81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae"}
Oct 30 20:31:14.525133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.525055    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:14.525133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.525079    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" event={"ID":"954ad950-625d-4ae9-b98f-9b8c54017993","Type":"ContainerStarted","Data":"14eb429d161a3a87056b0dbd863032e355f34ab35622f7ff822cb53ebbb6d648"}
Oct 30 20:31:14.659403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.659357    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" podStartSLOduration=1.6593445820000001 podStartE2EDuration="1.659344582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:14.65622208 +0000 UTC m=+655.257024039" watchObservedRunningTime="2026-10-30 20:31:14.659344582 +0000 UTC m=+655.260146541"
Oct 30 20:31:14.773779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:14.773555    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" podStartSLOduration=1.7735337310000001 podStartE2EDuration="1.773533731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:14.770667236 +0000 UTC m=+655.371469232" watchObservedRunningTime="2026-10-30 20:31:14.773533731 +0000 UTC m=+655.374335692"
Oct 30 20:31:18.488908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.488303    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"]
Oct 30 20:31:18.493143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.493117    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:18.547432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.547399    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"]
Oct 30 20:31:18.594379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.594355    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-4ee72d\""
Oct 30 20:31:18.594379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.594368    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-4ee72d-dockercfg-mn8rs\""
Oct 30 20:31:18.654559 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.654534    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:18.654682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.654653    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqh29\" (UniqueName: \"kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29\") pod \"payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:18.756072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.756018    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqh29\" (UniqueName: \"kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29\") pod \"payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:18.756072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.756069    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:18.756653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.756637    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:18.812684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:18.812663    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqh29\" (UniqueName: \"kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29\") pod \"payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:19.106940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:19.106866    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:19.479686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:19.479664    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"]
Oct 30 20:31:19.481849 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:19.481823    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3843b4e9_ba5b_4a9d_b53e_c3e9f04d44d6.slice/crio-752426cbb4257a56fbe2050ba48f7234c088fc1f2f22dcf2db6fa311ba06c51e WatchSource:0}: Error finding container 752426cbb4257a56fbe2050ba48f7234c088fc1f2f22dcf2db6fa311ba06c51e: Status 404 returned error can't find the container with id 752426cbb4257a56fbe2050ba48f7234c088fc1f2f22dcf2db6fa311ba06c51e
Oct 30 20:31:19.554756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:19.554710    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r" event={"ID":"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6","Type":"ContainerStarted","Data":"9675ffa245e6c8fc63d7f0a08974af3d9c35af84b40b9959836ff65f65f485c7"}
Oct 30 20:31:19.555121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:19.554788    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r" event={"ID":"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6","Type":"ContainerStarted","Data":"752426cbb4257a56fbe2050ba48f7234c088fc1f2f22dcf2db6fa311ba06c51e"}
Oct 30 20:31:19.555121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:19.554833    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:19.676992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:19.676951    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r" podStartSLOduration=1.676937533 podStartE2EDuration="1.676937533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:19.674928482 +0000 UTC m=+660.275730443" watchObservedRunningTime="2026-10-30 20:31:19.676937533 +0000 UTC m=+660.277739493"
Oct 30 20:31:20.880274 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.880237    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"]
Oct 30 20:31:20.885132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.885116    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:20.945868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.945838    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0a78ac\""
Oct 30 20:31:20.945868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.945838    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0a78ac-dockercfg-9n5nm\""
Oct 30 20:31:20.957830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.957803    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"]
Oct 30 20:31:20.979573 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.979542    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:20.979705 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:20.979617    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tsr\" (UniqueName: \"kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr\") pod \"payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:21.025580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.025531    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"]
Oct 30 20:31:21.032465 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.032423    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.080702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.080677    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:21.080868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.080776    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tsr\" (UniqueName: \"kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr\") pod \"payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:21.081393 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.081348    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:21.085722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.085693    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"]
Oct 30 20:31:21.137668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.137612    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tsr\" (UniqueName: \"kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr\") pod \"payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:21.182206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.182184    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.182322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.182247    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjld8\" (UniqueName: \"kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8\") pod \"payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.197984 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.197957    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:21.283288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.283254    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjld8\" (UniqueName: \"kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8\") pod \"payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.283456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.283438    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.284024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.284004    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.340556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.340526    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjld8\" (UniqueName: \"kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8\") pod \"payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.348326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.348271    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:21.568629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.568603    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"]
Oct 30 20:31:21.570644 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:21.570622    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9918a6dc_843b_483b_ab96_c50c676c13de.slice/crio-360298ed23a9f36b33e451a275629b346b1e636458361682ce7750a9721993e1 WatchSource:0}: Error finding container 360298ed23a9f36b33e451a275629b346b1e636458361682ce7750a9721993e1: Status 404 returned error can't find the container with id 360298ed23a9f36b33e451a275629b346b1e636458361682ce7750a9721993e1
Oct 30 20:31:21.735082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:21.735060    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"]
Oct 30 20:31:21.736962 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:21.736940    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb207bc_4523_44b8_88af_5c038f22a2ef.slice/crio-e2c480c9c569c655c2d80ec546a3a68b6f82d7c8591634978fb614381e407d85 WatchSource:0}: Error finding container e2c480c9c569c655c2d80ec546a3a68b6f82d7c8591634978fb614381e407d85: Status 404 returned error can't find the container with id e2c480c9c569c655c2d80ec546a3a68b6f82d7c8591634978fb614381e407d85
Oct 30 20:31:22.572582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.572543    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" event={"ID":"fbb207bc-4523-44b8-88af-5c038f22a2ef","Type":"ContainerStarted","Data":"70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181"}
Oct 30 20:31:22.573026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.572612    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:22.573026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.572634    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" event={"ID":"fbb207bc-4523-44b8-88af-5c038f22a2ef","Type":"ContainerStarted","Data":"e2c480c9c569c655c2d80ec546a3a68b6f82d7c8591634978fb614381e407d85"}
Oct 30 20:31:22.574838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.574804    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" event={"ID":"9918a6dc-843b-483b-ab96-c50c676c13de","Type":"ContainerStarted","Data":"ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c"}
Oct 30 20:31:22.574957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.574859    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" event={"ID":"9918a6dc-843b-483b-ab96-c50c676c13de","Type":"ContainerStarted","Data":"360298ed23a9f36b33e451a275629b346b1e636458361682ce7750a9721993e1"}
Oct 30 20:31:22.574957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.574887    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:22.690426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.690374    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" podStartSLOduration=2.690360003 podStartE2EDuration="2.690360003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:22.688035122 +0000 UTC m=+663.288837082" watchObservedRunningTime="2026-10-30 20:31:22.690360003 +0000 UTC m=+663.291161994"
Oct 30 20:31:22.806892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:22.806695    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" podStartSLOduration=2.806678938 podStartE2EDuration="2.806678938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:22.805920898 +0000 UTC m=+663.406722859" watchObservedRunningTime="2026-10-30 20:31:22.806678938 +0000 UTC m=+663.407480899"
Oct 30 20:31:25.533011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:25.532979    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 20:31:25.533391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:25.533150    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 20:31:28.430709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.430676    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"]
Oct 30 20:31:28.434913 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.434895    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.456646 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.456626    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.456769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.456693    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2gt\" (UniqueName: \"kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt\") pod \"payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.490628 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.490605    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5f9b95\""
Oct 30 20:31:28.490787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.490762    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5f9b95-dockercfg-8sr5g\""
Oct 30 20:31:28.542383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.542356    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"]
Oct 30 20:31:28.544338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.544302    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"]
Oct 30 20:31:28.548129 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.548110    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.557720 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.557694    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.557834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.557731    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.557834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.557824    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gxmn\" (UniqueName: \"kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn\") pod \"payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.557946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.557850    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2gt\" (UniqueName: \"kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt\") pod \"payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.558371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.558350    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.603629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.603604    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"]
Oct 30 20:31:28.614444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.614419    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2gt\" (UniqueName: \"kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt\") pod \"payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.660015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.659985    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.660125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.660095    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gxmn\" (UniqueName: \"kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn\") pod \"payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.661094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.661069    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.718110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.718064    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gxmn\" (UniqueName: \"kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn\") pod \"payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:28.747929 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.747911    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:28.860202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:28.860169    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:29.118344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.118321    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"]
Oct 30 20:31:29.120920 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:29.120895    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a64963_39b1_4577_b18e_50d8b4fa98c9.slice/crio-a2702b53bfb1368df3cb5122c8dec6e77f8c7a10dfaf893047ffaefb349feb3d WatchSource:0}: Error finding container a2702b53bfb1368df3cb5122c8dec6e77f8c7a10dfaf893047ffaefb349feb3d: Status 404 returned error can't find the container with id a2702b53bfb1368df3cb5122c8dec6e77f8c7a10dfaf893047ffaefb349feb3d
Oct 30 20:31:29.236611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.236591    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"]
Oct 30 20:31:29.238757 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:29.238719    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd52691_dd57_4252_a51d_3b1631bcf8a4.slice/crio-f2b0ab65311f61b51d3e1aa34a532154c668018a460d53929db02041020ee3ea WatchSource:0}: Error finding container f2b0ab65311f61b51d3e1aa34a532154c668018a460d53929db02041020ee3ea: Status 404 returned error can't find the container with id f2b0ab65311f61b51d3e1aa34a532154c668018a460d53929db02041020ee3ea
Oct 30 20:31:29.612184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.612153    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" event={"ID":"2bd52691-dd57-4252-a51d-3b1631bcf8a4","Type":"ContainerStarted","Data":"9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f"}
Oct 30 20:31:29.612581 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.612203    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" event={"ID":"2bd52691-dd57-4252-a51d-3b1631bcf8a4","Type":"ContainerStarted","Data":"f2b0ab65311f61b51d3e1aa34a532154c668018a460d53929db02041020ee3ea"}
Oct 30 20:31:29.612581 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.612232    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:29.613813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.613789    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" event={"ID":"59a64963-39b1-4577-b18e-50d8b4fa98c9","Type":"ContainerStarted","Data":"0435a2d9331ee04443a37260ef73594bd77db7246513fa880dcbe98ec3c8417a"}
Oct 30 20:31:29.613930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.613833    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" event={"ID":"59a64963-39b1-4577-b18e-50d8b4fa98c9","Type":"ContainerStarted","Data":"a2702b53bfb1368df3cb5122c8dec6e77f8c7a10dfaf893047ffaefb349feb3d"}
Oct 30 20:31:29.613930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.613864    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:29.726545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.726497    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" podStartSLOduration=1.726484753 podStartE2EDuration="1.726484753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:29.725984288 +0000 UTC m=+670.326786248" watchObservedRunningTime="2026-10-30 20:31:29.726484753 +0000 UTC m=+670.327286712"
Oct 30 20:31:29.841419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:29.841371    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" podStartSLOduration=1.841356555 podStartE2EDuration="1.841356555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:29.84017963 +0000 UTC m=+670.440981590" watchObservedRunningTime="2026-10-30 20:31:29.841356555 +0000 UTC m=+670.442158514"
Oct 30 20:31:30.563617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:30.563587    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 20:31:31.980780 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:31:31.980719    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:31:33.583175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:33.583146    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 20:31:33.583634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:33.583206    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 20:31:35.793766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.793128    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"]
Oct 30 20:31:35.798077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.798056    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:35.831385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.831359    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:35.831519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.831420    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gjd\" (UniqueName: \"kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd\") pod \"payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:35.853918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.853891    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8e2fc1\""
Oct 30 20:31:35.903527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.903500    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8e2fc1-dockercfg-65tpx\""
Oct 30 20:31:35.905507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.905484    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"]
Oct 30 20:31:35.933043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.933019    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:35.933169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.933073    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gjd\" (UniqueName: \"kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd\") pod \"payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:35.933656 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.933637    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:35.994358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:35.994328    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gjd\" (UniqueName: \"kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd\") pod \"payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:36.115911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:36.115836    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:36.502190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:36.502167    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"]
Oct 30 20:31:36.504451 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:36.504422    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac979096_2a91_4bdb_9e84_324839fcb845.slice/crio-72a28aaf7adbebf6abc418e72738ef6787e26cfea13ffa9c489faa53ac324f77 WatchSource:0}: Error finding container 72a28aaf7adbebf6abc418e72738ef6787e26cfea13ffa9c489faa53ac324f77: Status 404 returned error can't find the container with id 72a28aaf7adbebf6abc418e72738ef6787e26cfea13ffa9c489faa53ac324f77
Oct 30 20:31:36.651886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:36.651852    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" event={"ID":"ac979096-2a91-4bdb-9e84-324839fcb845","Type":"ContainerStarted","Data":"a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f"}
Oct 30 20:31:36.652063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:36.651912    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" event={"ID":"ac979096-2a91-4bdb-9e84-324839fcb845","Type":"ContainerStarted","Data":"72a28aaf7adbebf6abc418e72738ef6787e26cfea13ffa9c489faa53ac324f77"}
Oct 30 20:31:36.652063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:36.652052    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:36.767992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:36.767889    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" podStartSLOduration=1.767870987 podStartE2EDuration="1.767870987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:36.767014629 +0000 UTC m=+677.367816589" watchObservedRunningTime="2026-10-30 20:31:36.767870987 +0000 UTC m=+677.368672960"
Oct 30 20:31:38.361423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.361391    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"]
Oct 30 20:31:38.365776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.365730    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.418117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.418092    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ae2d38\""
Oct 30 20:31:38.461667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.461643    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bj6\" (UniqueName: \"kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6\") pod \"payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.461864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.461803    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.466976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.466955    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ae2d38-dockercfg-f56p2\""
Oct 30 20:31:38.473762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.470198    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"]
Oct 30 20:31:38.562587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.562552    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bj6\" (UniqueName: \"kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6\") pod \"payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.562757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.562654    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.563347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.563327    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.619089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.619027    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bj6\" (UniqueName: \"kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6\") pod \"payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:38.701503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:38.701472    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:39.079034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:39.079002    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"]
Oct 30 20:31:39.083850 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:31:39.083816    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc49c7ea_6e79_464e_9914_d3a42d35ea5d.slice/crio-9727cea8becf79047f694c022e0d9920ce58f0aa6d56013bc9eef147d3e10b3b WatchSource:0}: Error finding container 9727cea8becf79047f694c022e0d9920ce58f0aa6d56013bc9eef147d3e10b3b: Status 404 returned error can't find the container with id 9727cea8becf79047f694c022e0d9920ce58f0aa6d56013bc9eef147d3e10b3b
Oct 30 20:31:39.674454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:39.674420    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw" event={"ID":"fc49c7ea-6e79-464e-9914-d3a42d35ea5d","Type":"ContainerStarted","Data":"ab39175a936747356d3a6f3b91a28ee69f84e296903d9c4169e3576081dcfb82"}
Oct 30 20:31:39.674967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:39.674483    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:39.674967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:39.674504    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw" event={"ID":"fc49c7ea-6e79-464e-9914-d3a42d35ea5d","Type":"ContainerStarted","Data":"9727cea8becf79047f694c022e0d9920ce58f0aa6d56013bc9eef147d3e10b3b"}
Oct 30 20:31:39.791439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:39.791370    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw" podStartSLOduration=1.791349968 podStartE2EDuration="1.791349968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:39.790254217 +0000 UTC m=+680.391056186" watchObservedRunningTime="2026-10-30 20:31:39.791349968 +0000 UTC m=+680.392151929"
Oct 30 20:31:40.622980 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:40.622950    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:31:40.623173 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:40.623026    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:31:42.697495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:42.697387    5967 generic.go:356] "Generic (PLEG): container finished" podID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerID="bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f" exitCode=0
Oct 30 20:31:42.697495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:42.697442    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" event={"ID":"4fb7b62c-26d5-4657-b164-c1549cb4942d","Type":"ContainerDied","Data":"bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"}
Oct 30 20:31:42.698214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:42.697825    5967 scope.go:122] "RemoveContainer" containerID="bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"
Oct 30 20:31:42.699309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:42.699275    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:42.699441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:42.699366    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:42.699507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:42.699486    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:43.721033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:43.720992    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" event={"ID":"4fb7b62c-26d5-4657-b164-c1549cb4942d","Type":"ContainerStarted","Data":"e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c"}
Oct 30 20:31:44.217764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:44.217127    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:44.217764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:44.217253    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:31:44.217764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:44.217299    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 20:31:45.219810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:45.219730    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:31:45.220270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:45.219855    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 20:31:46.218158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:46.218122    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:31:46.218829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:46.218405    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 20:31:47.217210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:47.217169    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:31:47.217906 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:47.217843    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 20:31:47.665282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:47.665194    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 20:31:48.217598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:48.217560    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:31:48.218037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:48.217625    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 20:31:49.221254 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:49.221223    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:49.221729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:49.221619    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:49.222398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:49.222378    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 20:31:50.682974 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:50.682944    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 20:31:51.795835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:51.795803    5967 generic.go:356] "Generic (PLEG): container finished" podID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerID="f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304" exitCode=0
Oct 30 20:31:51.796376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:51.795884    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" event={"ID":"8324624e-fbd0-4fc1-92c1-05c60d635176","Type":"ContainerDied","Data":"f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"}
Oct 30 20:31:51.796376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:51.796366    5967 scope.go:122] "RemoveContainer" containerID="f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"
Oct 30 20:31:51.797680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:51.797652    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:51.797795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:51.797718    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:51.797795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:51.797763    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:52.490765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:52.489885    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:52.820988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:52.820897    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" event={"ID":"8324624e-fbd0-4fc1-92c1-05c60d635176","Type":"ContainerStarted","Data":"6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a"}
Oct 30 20:31:53.557049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:53.557002    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:53.557632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:53.557612    5967 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:31:53.557680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:53.557665    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 20:31:54.556836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:54.556801    5967 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:31:54.557215 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:54.556863    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 20:31:55.557423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:55.557390    5967 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:31:55.557833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:55.557438    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 20:31:56.557077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:56.557005    5967 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:31:56.557077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:56.557063    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 20:31:57.557197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:57.557153    5967 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:31:57.557597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:57.557224    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 20:31:58.561299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:58.561266    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:58.561779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:58.561554    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:31:58.562343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:31:58.562327    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 20:32:00.872536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:00.872504    5967 generic.go:356] "Generic (PLEG): container finished" podID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerID="26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46" exitCode=0
Oct 30 20:32:00.873016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:00.872579    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" event={"ID":"48f0e4ca-0fde-4823-a54d-72663bbf30ed","Type":"ContainerDied","Data":"26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"}
Oct 30 20:32:00.873060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:00.873025    5967 scope.go:122] "RemoveContainer" containerID="26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"
Oct 30 20:32:00.874488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:00.874447    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:00.874583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:00.874519    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:00.874583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:00.874562    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:01.905678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:01.905639    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" event={"ID":"48f0e4ca-0fde-4823-a54d-72663bbf30ed","Type":"ContainerStarted","Data":"fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd"}
Oct 30 20:32:02.002847 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:32:02.002799    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:32:02.626257 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:02.626225    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:32:02.626454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:02.626441    5967 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:32:02.626506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:02.626492    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 20:32:03.625766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:03.625689    5967 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:32:03.626147 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:03.625793    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 20:32:04.626055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:04.625935    5967 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:32:04.626055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:04.626014    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 20:32:05.626252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:05.626218    5967 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:32:05.626670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:05.626283    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 20:32:06.626340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:06.626305    5967 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:32:06.626787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:06.626364    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 20:32:07.631649 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:07.631617    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:32:07.632099 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:07.631934    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:32:07.632646 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:07.632630    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:32:20.034696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:20.034660    5967 generic.go:356] "Generic (PLEG): container finished" podID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerID="ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae" exitCode=0
Oct 30 20:32:20.035166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:20.034703    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" event={"ID":"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7","Type":"ContainerDied","Data":"ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"}
Oct 30 20:32:20.035166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:20.035133    5967 scope.go:122] "RemoveContainer" containerID="ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"
Oct 30 20:32:20.036481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:20.036454    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:20.036556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:20.036519    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:20.036556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:20.036549    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:21.043779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:21.043729    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" event={"ID":"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7","Type":"ContainerStarted","Data":"f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab"}
Oct 30 20:32:22.016839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:22.016799    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:32:22.018270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:22.018240    5967 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:32:22.018411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:22.018301    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 20:32:23.017222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:23.017178    5967 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:32:23.017638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:23.017238    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 20:32:23.493907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:23.493856    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:32:24.016825 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:24.016789    5967 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:32:24.017013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:24.016842    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 20:32:25.016827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:25.016789    5967 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:32:25.017251 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:25.016862    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 20:32:26.016798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:26.016716    5967 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:32:26.017025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:26.016825    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 20:32:27.021792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:27.021730    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:32:27.023925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:27.023900    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 20:32:31.981508 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:32:31.981426    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:32:39.724855 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.724813    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"]
Oct 30 20:32:39.728136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.728111    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.806378 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.806334    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.806557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.806509    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwfj\" (UniqueName: \"kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.806618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.806559    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.806733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.806715    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.806833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.806776    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.806890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.806832    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.817367 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.817339    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"default-dockercfg-4cgd2\""
Oct 30 20:32:39.818830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.818808    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"openshift-service-ca.crt\""
Oct 30 20:32:39.818972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.818834    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"simulator-w3-ae2d38-kserve-self-signed-certs\""
Oct 30 20:32:39.830889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.830862    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"kube-root-ca.crt\""
Oct 30 20:32:39.834130 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.834093    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"]
Oct 30 20:32:39.907802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.907763    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.907811    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.907830    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.907877    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.907947    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwfj\" (UniqueName: \"kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.907978    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.908273    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.908280    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.908431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.908345    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.910402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.910378    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.910572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.910553    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:39.968110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:39.968078    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwfj\" (UniqueName: \"kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj\") pod \"simulator-w3-ae2d38-kserve-646f8fd99-t4fxd\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") " pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:40.039033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:40.038929    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:40.424020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:40.423889    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"]
Oct 30 20:32:40.427103 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:32:40.427073    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6889cfe8_8a85_43c2_b07a_3902ac8d1d97.slice/crio-e106c3df1e25b7760b9bf9b1c71d5a89af680fc695b0addb65e16bb27eaca0b0 WatchSource:0}: Error finding container e106c3df1e25b7760b9bf9b1c71d5a89af680fc695b0addb65e16bb27eaca0b0: Status 404 returned error can't find the container with id e106c3df1e25b7760b9bf9b1c71d5a89af680fc695b0addb65e16bb27eaca0b0
Oct 30 20:32:41.164812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:41.164777    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" event={"ID":"6889cfe8-8a85-43c2-b07a-3902ac8d1d97","Type":"ContainerStarted","Data":"3524f0c22bc7bf5d870c74c33dbe365b4d57157b6ee060db8fe561d794abf9d1"}
Oct 30 20:32:41.165229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:41.164820    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" event={"ID":"6889cfe8-8a85-43c2-b07a-3902ac8d1d97","Type":"ContainerStarted","Data":"e106c3df1e25b7760b9bf9b1c71d5a89af680fc695b0addb65e16bb27eaca0b0"}
Oct 30 20:32:41.291117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:41.291049    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podStartSLOduration=2.291032784 podStartE2EDuration="2.291032784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:41.290891955 +0000 UTC m=+741.891693919" watchObservedRunningTime="2026-10-30 20:32:41.291032784 +0000 UTC m=+741.891834746"
Oct 30 20:32:42.039445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:42.039355    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:50.040105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:50.040071    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:50.053460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:50.053428    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:32:50.231093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:32:50.231062    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:33:01.117066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.117023    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"]
Oct 30 20:33:01.120246 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.120212    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.194901 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.194865    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"openshift-service-ca.crt\""
Oct 30 20:33:01.225356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.225323    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"kube-root-ca.crt\""
Oct 30 20:33:01.225567 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.225425    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"default-dockercfg-999rs\""
Oct 30 20:33:01.225567 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.225545    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"simulator-w5-a78bf4-kserve-self-signed-certs\""
Oct 30 20:33:01.227774 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.227726    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"]
Oct 30 20:33:01.241080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.241047    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8btp\" (UniqueName: \"kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.241239 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.241099    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.241239 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.241196    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.241318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.241240    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.241318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.241264    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.241380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.241351    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.342582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.342547    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.342802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.342597    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.342802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.342711    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.342802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.342783    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.342947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.342915    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8btp\" (UniqueName: \"kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.343004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.342963    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.343088 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.343060    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.343231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.343115    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.343497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.343395    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.345056 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.345038    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.345333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.345314    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.402409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.402326    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8btp\" (UniqueName: \"kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp\") pod \"simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") " pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.432771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.432703    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:01.812394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:01.812364    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"]
Oct 30 20:33:01.814168 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:33:01.814137    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaf85b46_827c_4f06_a493_e7c47c1c3681.slice/crio-be2fdd8ba8630618fc64302b6028a5925a91004647a2721302ddcf3bfd080d5b WatchSource:0}: Error finding container be2fdd8ba8630618fc64302b6028a5925a91004647a2721302ddcf3bfd080d5b: Status 404 returned error can't find the container with id be2fdd8ba8630618fc64302b6028a5925a91004647a2721302ddcf3bfd080d5b
Oct 30 20:33:01.981186 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:33:01.981143    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:33:02.291349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:02.291313    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" event={"ID":"daf85b46-827c-4f06-a493-e7c47c1c3681","Type":"ContainerStarted","Data":"f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94"}
Oct 30 20:33:02.291868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:02.291358    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" event={"ID":"daf85b46-827c-4f06-a493-e7c47c1c3681","Type":"ContainerStarted","Data":"be2fdd8ba8630618fc64302b6028a5925a91004647a2721302ddcf3bfd080d5b"}
Oct 30 20:33:02.406250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:02.406196    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podStartSLOduration=1.406180255 podStartE2EDuration="1.406180255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:02.405290451 +0000 UTC m=+763.006092412" watchObservedRunningTime="2026-10-30 20:33:02.406180255 +0000 UTC m=+763.006982215"
Oct 30 20:33:02.433527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:02.433490    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:11.433194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:11.433158    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:11.446001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:11.445973    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:12.361776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:12.361724    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:33:31.981213 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:33:31.981173    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:33:43.655666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.655630    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"]
Oct 30 20:33:43.659577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.659557    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.760887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.760812    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"kube-root-ca.crt\""
Oct 30 20:33:43.760887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.760824    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"openshift-service-ca.crt\""
Oct 30 20:33:43.761104 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.760897    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"premium-w0-0a78ac-kserve-self-signed-certs\""
Oct 30 20:33:43.761104 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.760975    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"default-dockercfg-h5hv9\""
Oct 30 20:33:43.763066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.763048    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"]
Oct 30 20:33:43.804356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.804314    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.804558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.804427    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.804558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.804460    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.804558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.804518    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gv5\" (UniqueName: \"kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.804730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.804583    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.804730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.804628    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.905805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.905690    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.905805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.905785    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gv5\" (UniqueName: \"kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.905833    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.905859    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.905896    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.905956    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.906298    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.906345    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.906375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.906361    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.908311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.908280    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.908697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.908677    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.962679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.962639    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gv5\" (UniqueName: \"kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5\") pod \"premium-w0-0a78ac-kserve-7f79767f4b-ljvvs\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") " pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:43.971711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:43.971666    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:44.365588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:44.365555    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"]
Oct 30 20:33:44.367582 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:33:44.367555    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd683f1_7b7d_499a_b774_cce9364fb71a.slice/crio-be7a9a1eb523231f1bce055c1608a1a5678ad7d86b6d1ac28b82a4ce13b654b7 WatchSource:0}: Error finding container be7a9a1eb523231f1bce055c1608a1a5678ad7d86b6d1ac28b82a4ce13b654b7: Status 404 returned error can't find the container with id be7a9a1eb523231f1bce055c1608a1a5678ad7d86b6d1ac28b82a4ce13b654b7
Oct 30 20:33:44.547222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:44.547180    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" event={"ID":"0fd683f1-7b7d-499a-b774-cce9364fb71a","Type":"ContainerStarted","Data":"316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e"}
Oct 30 20:33:44.547400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:44.547233    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" event={"ID":"0fd683f1-7b7d-499a-b774-cce9364fb71a","Type":"ContainerStarted","Data":"be7a9a1eb523231f1bce055c1608a1a5678ad7d86b6d1ac28b82a4ce13b654b7"}
Oct 30 20:33:44.664188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:44.664063    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podStartSLOduration=1.6640463429999999 podStartE2EDuration="1.664046343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:44.661908914 +0000 UTC m=+805.262710875" watchObservedRunningTime="2026-10-30 20:33:44.664046343 +0000 UTC m=+805.264848303"
Oct 30 20:33:44.971847 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:44.971814    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:53.972245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:53.972203    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:53.986370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:53.986348    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:33:54.616309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:33:54.616278    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:34:01.981312 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:34:01.981224    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:34:21.173793 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.173755    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"]
Oct 30 20:34:21.177885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.177865    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.229624 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.229584    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-4ee72d-openshift-default-dockercfg-5dldm\""
Oct 30 20:34:21.281593 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.281557    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"]
Oct 30 20:34:21.329388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329352    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329390    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329562    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329620    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329701    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329723    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7t2c\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329771    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329873    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.329921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.329899    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.430878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.430776    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.430878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.430818    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.430878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.430838    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.430878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.430860    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.430936    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.430982    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431037    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431064    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7t2c\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431101    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431229    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431243    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431318    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431519    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.431722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.431700    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.433448 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.433421    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.433669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.433649    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.487373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.487337    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.487523 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.487476    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7t2c\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.491314 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.491297    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:21.923284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.923262    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"]
Oct 30 20:34:21.925304 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:34:21.925267    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a10ecd_1f2d_46c6_a4b7_b9aebc7d4232.slice/crio-c5417affdaa4e8dd2d6848df22f6bc0f34225a70ba26a1847adb5ea3dd390f05 WatchSource:0}: Error finding container c5417affdaa4e8dd2d6848df22f6bc0f34225a70ba26a1847adb5ea3dd390f05: Status 404 returned error can't find the container with id c5417affdaa4e8dd2d6848df22f6bc0f34225a70ba26a1847adb5ea3dd390f05
Oct 30 20:34:21.927804 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.927770    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:34:21.927890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.927836    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:34:21.927890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:21.927865    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:34:22.770127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:22.770091    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" event={"ID":"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232","Type":"ContainerStarted","Data":"f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156"}
Oct 30 20:34:22.790766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:22.770704    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" event={"ID":"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232","Type":"ContainerStarted","Data":"c5417affdaa4e8dd2d6848df22f6bc0f34225a70ba26a1847adb5ea3dd390f05"}
Oct 30 20:34:23.497982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:23.497930    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:23.499500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:23.499469    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:34:23.499662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:23.499525    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:34:24.494767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:24.494585    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:34:24.494767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:24.494660    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:34:25.495131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:25.495088    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:34:25.495657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:25.495160    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:34:26.494178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:26.494142    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:34:26.494448 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:26.494213    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:34:27.492768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:27.492201    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:34:27.492768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:27.492309    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:34:28.492157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:28.492120    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:34:28.492351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:28.492180    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:34:29.496731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:29.496700    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:29.497184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:29.497090    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:29.497795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:29.497776    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:34:29.663385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:29.663318    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podStartSLOduration=8.663302811 podStartE2EDuration="8.663302811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:22.962285868 +0000 UTC m=+843.563087826" watchObservedRunningTime="2026-10-30 20:34:29.663302811 +0000 UTC m=+850.264104772"
Oct 30 20:34:31.980984 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:34:31.980948    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:34:32.408812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.408719    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"]
Oct 30 20:34:32.412800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.412775    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.472504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.472473    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.472638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.472526    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.472678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.472659    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrd5j\" (UniqueName: \"kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.516382 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.516338    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:32.516382 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.516338    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:32.516382 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.516346    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:32.516382 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.516356    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:34:32.518766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.518726    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"]
Oct 30 20:34:32.573502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.573474    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrd5j\" (UniqueName: \"kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.573644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.573561    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.573644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.573591    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.574198 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.574177    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.575894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.575879    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.630184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.630159    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrd5j\" (UniqueName: \"kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j\") pod \"maas-api-revoke-sub-9b52ab98-h48p5\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:32.726372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:32.726337    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:33.129486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:33.129460    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"]
Oct 30 20:34:33.131006 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:34:33.130976    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda903e2_dd7c_4f1d_b4c7_d878affd0e72.slice/crio-40d2810d75ff92d3993429979329ef9edfa9c161307b31d346fe06807329d830 WatchSource:0}: Error finding container 40d2810d75ff92d3993429979329ef9edfa9c161307b31d346fe06807329d830: Status 404 returned error can't find the container with id 40d2810d75ff92d3993429979329ef9edfa9c161307b31d346fe06807329d830
Oct 30 20:34:33.876116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:33.876083    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5" event={"ID":"cda903e2-dd7c-4f1d-b4c7-d878affd0e72","Type":"ContainerStarted","Data":"4fc1776e5e4056220297f7578f01b4aa6f54507adca747a11a2997095320bccb"}
Oct 30 20:34:33.876285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:33.876152    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5" event={"ID":"cda903e2-dd7c-4f1d-b4c7-d878affd0e72","Type":"ContainerStarted","Data":"40d2810d75ff92d3993429979329ef9edfa9c161307b31d346fe06807329d830"}
Oct 30 20:34:33.991329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:33.991273    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5" podStartSLOduration=1.991258226 podStartE2EDuration="1.991258226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:33.99008001 +0000 UTC m=+854.590881968" watchObservedRunningTime="2026-10-30 20:34:33.991258226 +0000 UTC m=+854.592060185"
Oct 30 20:34:34.882920 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:34.882880    5967 generic.go:356] "Generic (PLEG): container finished" podID="cda903e2-dd7c-4f1d-b4c7-d878affd0e72" containerID="4fc1776e5e4056220297f7578f01b4aa6f54507adca747a11a2997095320bccb" exitCode=0
Oct 30 20:34:34.883385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:34.882958    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5" event={"ID":"cda903e2-dd7c-4f1d-b4c7-d878affd0e72","Type":"ContainerDied","Data":"4fc1776e5e4056220297f7578f01b4aa6f54507adca747a11a2997095320bccb"}
Oct 30 20:34:36.116428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.116398    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:36.120182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.120148    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j\" (UniqueName: \"kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j\") pod \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") "
Oct 30 20:34:36.120349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.120199    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca\") pod \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") "
Oct 30 20:34:36.120412 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.120371    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca\") pod \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\" (UID: \"cda903e2-dd7c-4f1d-b4c7-d878affd0e72\") "
Oct 30 20:34:36.120913 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.120885    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca" pod "cda903e2-dd7c-4f1d-b4c7-d878affd0e72" (UID: "cda903e2-dd7c-4f1d-b4c7-d878affd0e72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:36.123044 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.123014    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca" pod "cda903e2-dd7c-4f1d-b4c7-d878affd0e72" (UID: "cda903e2-dd7c-4f1d-b4c7-d878affd0e72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:36.123154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.123112    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j" pod "cda903e2-dd7c-4f1d-b4c7-d878affd0e72" (UID: "cda903e2-dd7c-4f1d-b4c7-d878affd0e72"). InnerVolumeSpecName "kube-api-access-jrd5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:36.221459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.221420    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:36.221459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.221446    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:36.221459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.221459    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrd5j\" (UniqueName: \"kubernetes.io/projected/cda903e2-dd7c-4f1d-b4c7-d878affd0e72-kube-api-access-jrd5j\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:36.901004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.900967    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"
Oct 30 20:34:36.901177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.900963    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5" event={"ID":"cda903e2-dd7c-4f1d-b4c7-d878affd0e72","Type":"ContainerDied","Data":"40d2810d75ff92d3993429979329ef9edfa9c161307b31d346fe06807329d830"}
Oct 30 20:34:36.901177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:36.901090    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d2810d75ff92d3993429979329ef9edfa9c161307b31d346fe06807329d830"
Oct 30 20:34:47.864483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.864451    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"]
Oct 30 20:34:47.865046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.865032    5967 cpu_manager.go:414] "containerMap: removing container" podUID="cda903e2-dd7c-4f1d-b4c7-d878affd0e72" containerName="revoke-keys"
Oct 30 20:34:47.865091 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.865047    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda903e2-dd7c-4f1d-b4c7-d878affd0e72" containerName="revoke-keys"
Oct 30 20:34:47.865140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.865131    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda903e2-dd7c-4f1d-b4c7-d878affd0e72" containerName="revoke-keys"
Oct 30 20:34:47.868516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.868499    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:47.925870 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.925845    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"]
Oct 30 20:34:47.948680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.948657    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:47.948857 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.948823    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:47.948963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.948941    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grhz\" (UniqueName: \"kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:47.972189 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.972159    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:47.972295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.972195    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:47.972295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.972206    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:34:47.972295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:47.972206    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:48.049631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.049606    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.049786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.049650    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grhz\" (UniqueName: \"kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.049786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.049707    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.050343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.050311    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.051995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.051977    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.107141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.107118    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grhz\" (UniqueName: \"kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz\") pod \"maas-api-revoke-sub-b2707831-zdncc\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.181587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.181512    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:48.559050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.559021    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"]
Oct 30 20:34:48.560543 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:34:48.560517    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8769f3_120f_49bf_9f4c_87a3a9f65b06.slice/crio-16bfcd44cd2a48e0770e018a5d9208fbc2f0f0cccb19f497a66588a4abded0b2 WatchSource:0}: Error finding container 16bfcd44cd2a48e0770e018a5d9208fbc2f0f0cccb19f497a66588a4abded0b2: Status 404 returned error can't find the container with id 16bfcd44cd2a48e0770e018a5d9208fbc2f0f0cccb19f497a66588a4abded0b2
Oct 30 20:34:48.969983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.969950    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc" event={"ID":"1a8769f3-120f-49bf-9f4c-87a3a9f65b06","Type":"ContainerStarted","Data":"521fda86973eafc41b8642ed6aa5697f422a2613d57d5bae24a963ab657540f6"}
Oct 30 20:34:48.970372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:48.970009    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc" event={"ID":"1a8769f3-120f-49bf-9f4c-87a3a9f65b06","Type":"ContainerStarted","Data":"16bfcd44cd2a48e0770e018a5d9208fbc2f0f0cccb19f497a66588a4abded0b2"}
Oct 30 20:34:49.086012 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:49.085949    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc" podStartSLOduration=2.085922716 podStartE2EDuration="2.085922716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:49.084427777 +0000 UTC m=+869.685229738" watchObservedRunningTime="2026-10-30 20:34:49.085922716 +0000 UTC m=+869.686724676"
Oct 30 20:34:49.977072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:49.977040    5967 generic.go:356] "Generic (PLEG): container finished" podID="1a8769f3-120f-49bf-9f4c-87a3a9f65b06" containerID="521fda86973eafc41b8642ed6aa5697f422a2613d57d5bae24a963ab657540f6" exitCode=0
Oct 30 20:34:49.977431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:49.977120    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc" event={"ID":"1a8769f3-120f-49bf-9f4c-87a3a9f65b06","Type":"ContainerDied","Data":"521fda86973eafc41b8642ed6aa5697f422a2613d57d5bae24a963ab657540f6"}
Oct 30 20:34:51.133297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.133274    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:51.185365 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.185337    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca\") pod \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") "
Oct 30 20:34:51.185530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.185424    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz\" (UniqueName: \"kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz\") pod \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") "
Oct 30 20:34:51.185530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.185456    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca\") pod \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\" (UID: \"1a8769f3-120f-49bf-9f4c-87a3a9f65b06\") "
Oct 30 20:34:51.185667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.185647    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca" pod "1a8769f3-120f-49bf-9f4c-87a3a9f65b06" (UID: "1a8769f3-120f-49bf-9f4c-87a3a9f65b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:51.187579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.187558    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz" pod "1a8769f3-120f-49bf-9f4c-87a3a9f65b06" (UID: "1a8769f3-120f-49bf-9f4c-87a3a9f65b06"). InnerVolumeSpecName "kube-api-access-7grhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:51.187579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.187559    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca" pod "1a8769f3-120f-49bf-9f4c-87a3a9f65b06" (UID: "1a8769f3-120f-49bf-9f4c-87a3a9f65b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:51.286362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.286297    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:51.286362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.286323    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grhz\" (UniqueName: \"kubernetes.io/projected/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-kube-api-access-7grhz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:51.286362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.286333    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8769f3-120f-49bf-9f4c-87a3a9f65b06-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:51.990786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.990724    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc" event={"ID":"1a8769f3-120f-49bf-9f4c-87a3a9f65b06","Type":"ContainerDied","Data":"16bfcd44cd2a48e0770e018a5d9208fbc2f0f0cccb19f497a66588a4abded0b2"}
Oct 30 20:34:51.990786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.990783    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"
Oct 30 20:34:51.990995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:51.990810    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16bfcd44cd2a48e0770e018a5d9208fbc2f0f0cccb19f497a66588a4abded0b2"
Oct 30 20:34:55.609869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.609833    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"]
Oct 30 20:34:55.610437 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.610418    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1a8769f3-120f-49bf-9f4c-87a3a9f65b06" containerName="revoke-keys"
Oct 30 20:34:55.610437 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.610435    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8769f3-120f-49bf-9f4c-87a3a9f65b06" containerName="revoke-keys"
Oct 30 20:34:55.610633 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.610593    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8769f3-120f-49bf-9f4c-87a3a9f65b06" containerName="revoke-keys"
Oct 30 20:34:55.614228 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.614209    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.718966 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.718934    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"kube-root-ca.crt\""
Oct 30 20:34:55.719126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.718996    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"premium-w4-8e2fc1-kserve-self-signed-certs\""
Oct 30 20:34:55.719175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.719155    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"default-dockercfg-9824w\""
Oct 30 20:34:55.719269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.719255    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"openshift-service-ca.crt\""
Oct 30 20:34:55.721483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.721454    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"]
Oct 30 20:34:55.736243 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.736214    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.736387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.736273    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.736387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.736371    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.736466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.736401    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjrv\" (UniqueName: \"kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.736466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.736424    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.736593 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.736572    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.837817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.837759    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.837846    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.837887    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.837911    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjrv\" (UniqueName: \"kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.838038    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.838127    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838407 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.838383    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.838442    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.838554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.838527    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.840196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.840168    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.840429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.840412    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.894829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.894760    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjrv\" (UniqueName: \"kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv\") pod \"premium-w4-8e2fc1-kserve-847dcb587-c5sl7\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") " pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:55.927371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:55.927339    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:34:56.311560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:56.311538    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"]
Oct 30 20:34:56.313828 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:34:56.313798    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c5b9238_e7d3_4fc3_b731_aed5b4df138d.slice/crio-fb3b1744903ea9d570dad76cb56a3e74a6e8a9a7b2e641027d32765e2cdc22e6 WatchSource:0}: Error finding container fb3b1744903ea9d570dad76cb56a3e74a6e8a9a7b2e641027d32765e2cdc22e6: Status 404 returned error can't find the container with id fb3b1744903ea9d570dad76cb56a3e74a6e8a9a7b2e641027d32765e2cdc22e6
Oct 30 20:34:57.022285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:57.022246    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" event={"ID":"4c5b9238-e7d3-4fc3-b731-aed5b4df138d","Type":"ContainerStarted","Data":"1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a"}
Oct 30 20:34:57.022820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:57.022295    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" event={"ID":"4c5b9238-e7d3-4fc3-b731-aed5b4df138d","Type":"ContainerStarted","Data":"fb3b1744903ea9d570dad76cb56a3e74a6e8a9a7b2e641027d32765e2cdc22e6"}
Oct 30 20:34:57.138879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:57.138820    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podStartSLOduration=2.13880532 podStartE2EDuration="2.13880532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:57.137354074 +0000 UTC m=+877.738156036" watchObservedRunningTime="2026-10-30 20:34:57.13880532 +0000 UTC m=+877.739607279"
Oct 30 20:34:57.927529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:34:57.927483    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:35:01.979534 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:35:01.979493    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:35:03.623242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.623212    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"]
Oct 30 20:35:03.627358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.627337    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.682952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.682926    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:03.685771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.685729    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"]
Oct 30 20:35:03.728457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.728416    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.728904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.728876    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.729022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.728946    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghwn\" (UniqueName: \"kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.731660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.731640    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:03.731793 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.731724    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:03.731793 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.731785    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:35:03.830342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.830312    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghwn\" (UniqueName: \"kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.830512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.830418    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.830512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.830473    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.831117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.831090    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.832896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.832877    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.887232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.887166    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghwn\" (UniqueName: \"kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn\") pod \"maas-api-revoke-sub-25d42669-8xz7n\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:03.940989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:03.940954    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:04.318619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:04.318589    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"]
Oct 30 20:35:04.320110 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:04.320080    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e974644_61c0_47ca_85be_1543a960ba83.slice/crio-e74aaedfe2c097f2bd3a233340b8a701330a4874168f45f40dc33333d8301834 WatchSource:0}: Error finding container e74aaedfe2c097f2bd3a233340b8a701330a4874168f45f40dc33333d8301834: Status 404 returned error can't find the container with id e74aaedfe2c097f2bd3a233340b8a701330a4874168f45f40dc33333d8301834
Oct 30 20:35:05.069606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:05.069565    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n" event={"ID":"5e974644-61c0-47ca-85be-1543a960ba83","Type":"ContainerStarted","Data":"863ec317aab14f821738df4ca65ebb83c67def27bfa77c555a91ebdf3d89fef5"}
Oct 30 20:35:05.070014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:05.069637    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n" event={"ID":"5e974644-61c0-47ca-85be-1543a960ba83","Type":"ContainerStarted","Data":"e74aaedfe2c097f2bd3a233340b8a701330a4874168f45f40dc33333d8301834"}
Oct 30 20:35:05.187237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:05.187193    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n" podStartSLOduration=2.187178942 podStartE2EDuration="2.187178942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:05.186341246 +0000 UTC m=+885.787143206" watchObservedRunningTime="2026-10-30 20:35:05.187178942 +0000 UTC m=+885.787980902"
Oct 30 20:35:05.928410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:05.928379    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:35:05.940942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:05.940917    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:35:06.076452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:06.076426    5967 generic.go:356] "Generic (PLEG): container finished" podID="5e974644-61c0-47ca-85be-1543a960ba83" containerID="863ec317aab14f821738df4ca65ebb83c67def27bfa77c555a91ebdf3d89fef5" exitCode=0
Oct 30 20:35:06.076960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:06.076519    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n" event={"ID":"5e974644-61c0-47ca-85be-1543a960ba83","Type":"ContainerDied","Data":"863ec317aab14f821738df4ca65ebb83c67def27bfa77c555a91ebdf3d89fef5"}
Oct 30 20:35:06.088086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:06.088066    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:35:07.228931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.228909    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:07.368953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.368871    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca\") pod \"5e974644-61c0-47ca-85be-1543a960ba83\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") "
Oct 30 20:35:07.369119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.368959    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn\" (UniqueName: \"kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn\") pod \"5e974644-61c0-47ca-85be-1543a960ba83\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") "
Oct 30 20:35:07.369119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.369020    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca\") pod \"5e974644-61c0-47ca-85be-1543a960ba83\" (UID: \"5e974644-61c0-47ca-85be-1543a960ba83\") "
Oct 30 20:35:07.369408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.369382    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca" pod "5e974644-61c0-47ca-85be-1543a960ba83" (UID: "5e974644-61c0-47ca-85be-1543a960ba83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:07.371062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.371039    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn" pod "5e974644-61c0-47ca-85be-1543a960ba83" (UID: "5e974644-61c0-47ca-85be-1543a960ba83"). InnerVolumeSpecName "kube-api-access-2ghwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:07.371157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.371046    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca" pod "5e974644-61c0-47ca-85be-1543a960ba83" (UID: "5e974644-61c0-47ca-85be-1543a960ba83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:07.472265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.472231    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghwn\" (UniqueName: \"kubernetes.io/projected/5e974644-61c0-47ca-85be-1543a960ba83-kube-api-access-2ghwn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:07.472465 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.472278    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:07.472465 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:07.472296    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e974644-61c0-47ca-85be-1543a960ba83-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:08.093464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:08.093433    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n" event={"ID":"5e974644-61c0-47ca-85be-1543a960ba83","Type":"ContainerDied","Data":"e74aaedfe2c097f2bd3a233340b8a701330a4874168f45f40dc33333d8301834"}
Oct 30 20:35:08.093668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:08.093487    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e74aaedfe2c097f2bd3a233340b8a701330a4874168f45f40dc33333d8301834"
Oct 30 20:35:08.093668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:08.093533    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"
Oct 30 20:35:22.291672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.291639    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"]
Oct 30 20:35:22.292265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.292245    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5e974644-61c0-47ca-85be-1543a960ba83" containerName="revoke-keys"
Oct 30 20:35:22.292265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.292264    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e974644-61c0-47ca-85be-1543a960ba83" containerName="revoke-keys"
Oct 30 20:35:22.292417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.292360    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e974644-61c0-47ca-85be-1543a960ba83" containerName="revoke-keys"
Oct 30 20:35:22.296852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.296820    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.348947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.348927    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ae2d38-openshift-default-dockercfg-9z4cf\""
Oct 30 20:35:22.400958 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.400932    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"]
Oct 30 20:35:22.430398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430361    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430407    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430498    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430535    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430646    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz29c\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430683    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430865 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430800    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430866    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.430975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.430932    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.447090 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.447061    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"]
Oct 30 20:35:22.451452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.451434    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.503331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.503305    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f9b95-openshift-default-dockercfg-kpgjw\""
Oct 30 20:35:22.511315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.511289    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"]
Oct 30 20:35:22.531541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531516    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.531690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531551    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.531690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531631    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.531690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531660    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.531861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531698    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.531861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531828    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.531954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531877    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.532005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.531960    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.532052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532026    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532057    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532082    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532100    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.532195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532132    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532161    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532274    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz29c\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532315    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.532389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532330    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532365    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532414    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9jw\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.532582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532470    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532723    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.532879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.532800    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.533424 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.533400    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.534545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.534524    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.535008 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.534988    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.589585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.589530    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.589732 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.589710    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz29c\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.610024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.609992    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:22.633782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.633731    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.633950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.633915    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.633975    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634003    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634130 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634082    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634135    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634170    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634221    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9jw\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634275    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634299    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634425    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634599 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634584    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634621    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.634796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.634730    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.636729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.636708    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.637105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.637089    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.691300 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.691025    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.691300 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.691075    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9jw\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:22.764315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:22.764287    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:23.036088 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.036058    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"]
Oct 30 20:35:23.051305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.051272    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:23.051404 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.051335    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:23.051404 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.051365    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:23.147275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.146532    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"]
Oct 30 20:35:23.159879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.159828    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:23.160013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.159918    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:23.160013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.159966    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:23.185877 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.185834    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" event={"ID":"2c3c959b-7b4f-4cbd-84af-904623075aa0","Type":"ContainerStarted","Data":"5bb183e1df76a25046e38b1720d643644264a988fa385097b841df6d56648eba"}
Oct 30 20:35:23.187601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.187574    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" event={"ID":"42bf4655-d839-4ac7-a0bc-6cedc767c039","Type":"ContainerStarted","Data":"a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9"}
Oct 30 20:35:23.187811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.187776    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" event={"ID":"42bf4655-d839-4ac7-a0bc-6cedc767c039","Type":"ContainerStarted","Data":"e1f2635ed5213c01a7084ae34cc9dcab1da59df96c15f0b1e088db44e4463f7c"}
Oct 30 20:35:23.610768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.610521    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:23.963979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.963877    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podStartSLOduration=1.963862166 podStartE2EDuration="1.963862166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:23.310375702 +0000 UTC m=+903.911177660" watchObservedRunningTime="2026-10-30 20:35:23.963862166 +0000 UTC m=+904.564664129"
Oct 30 20:35:23.964472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.964454    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"]
Oct 30 20:35:23.968615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:23.968601    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.029765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.029309    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:24.041765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.034096    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"]
Oct 30 20:35:24.059352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.057835    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hklz\" (UniqueName: \"kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.059352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.057925    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.059352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.058024    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.111903 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.108785    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:24.127773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.113010    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:24.127997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.112817    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:35:24.159872 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.159080    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.159872 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.159231    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hklz\" (UniqueName: \"kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.159872 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.159315    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.161467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.160958    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.234976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.180313    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.261158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.261130    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hklz\" (UniqueName: \"kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz\") pod \"maas-api-revoke-sub-3a3d56b2-nznx4\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.267291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.267259    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" event={"ID":"2c3c959b-7b4f-4cbd-84af-904623075aa0","Type":"ContainerStarted","Data":"12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089"}
Oct 30 20:35:24.308764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.307947    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:24.622351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.622236    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:35:24.622351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.622286    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:35:24.765435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.765402    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:24.766800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.766780    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:35:24.766877 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.766824    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:35:24.791765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.785924    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podStartSLOduration=2.78590535 podStartE2EDuration="2.78590535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:24.401165382 +0000 UTC m=+905.001967344" watchObservedRunningTime="2026-10-30 20:35:24.78590535 +0000 UTC m=+905.386707313"
Oct 30 20:35:24.806765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:24.803415    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"]
Oct 30 20:35:24.824622 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:24.824593    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acfdf56_24b9_41eb_b76a_bd3bbea1f323.slice/crio-7632ac4b33cb39f5f1a44250d5813290a1a8e17879c221eefc8e353a851006bf WatchSource:0}: Error finding container 7632ac4b33cb39f5f1a44250d5813290a1a8e17879c221eefc8e353a851006bf: Status 404 returned error can't find the container with id 7632ac4b33cb39f5f1a44250d5813290a1a8e17879c221eefc8e353a851006bf
Oct 30 20:35:25.307764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:25.306793    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4" event={"ID":"8acfdf56-24b9-41eb-b76a-bd3bbea1f323","Type":"ContainerStarted","Data":"7632ac4b33cb39f5f1a44250d5813290a1a8e17879c221eefc8e353a851006bf"}
Oct 30 20:35:25.616845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:25.614101    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:35:25.616845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:25.614161    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:35:25.764923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:25.764887    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:35:25.765293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:25.764943    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:35:26.326885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:26.326793    5967 generic.go:356] "Generic (PLEG): container finished" podID="8acfdf56-24b9-41eb-b76a-bd3bbea1f323" containerID="98a9e06c6feead5ba0b4e6ddf8b518d2108b265154b977564b535598b4b910ec" exitCode=0
Oct 30 20:35:26.326885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:26.326869    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4" event={"ID":"8acfdf56-24b9-41eb-b76a-bd3bbea1f323","Type":"ContainerDied","Data":"98a9e06c6feead5ba0b4e6ddf8b518d2108b265154b977564b535598b4b910ec"}
Oct 30 20:35:26.619822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:26.615052    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:35:26.619822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:26.615111    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:35:26.764820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:26.764784    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:35:26.765043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:26.765019    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:35:27.562820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.562798    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:27.619436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.617060    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:35:27.619436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.618914    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:35:27.653256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.653218    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz\" (UniqueName: \"kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz\") pod \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") "
Oct 30 20:35:27.653421 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.653270    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca\") pod \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") "
Oct 30 20:35:27.653421 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.653373    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca\") pod \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\" (UID: \"8acfdf56-24b9-41eb-b76a-bd3bbea1f323\") "
Oct 30 20:35:27.654282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.654244    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca" pod "8acfdf56-24b9-41eb-b76a-bd3bbea1f323" (UID: "8acfdf56-24b9-41eb-b76a-bd3bbea1f323"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:27.656126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.656098    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca" pod "8acfdf56-24b9-41eb-b76a-bd3bbea1f323" (UID: "8acfdf56-24b9-41eb-b76a-bd3bbea1f323"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:27.656126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.656104    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz" pod "8acfdf56-24b9-41eb-b76a-bd3bbea1f323" (UID: "8acfdf56-24b9-41eb-b76a-bd3bbea1f323"). InnerVolumeSpecName "kube-api-access-8hklz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:27.754921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.754891    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hklz\" (UniqueName: \"kubernetes.io/projected/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-kube-api-access-8hklz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:27.754921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.754919    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:27.754921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.754929    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfdf56-24b9-41eb-b76a-bd3bbea1f323-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:27.765685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.765659    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:35:27.766036 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:27.765711    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:35:28.340903 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.340876    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"
Oct 30 20:35:28.341075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.340877    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4" event={"ID":"8acfdf56-24b9-41eb-b76a-bd3bbea1f323","Type":"ContainerDied","Data":"7632ac4b33cb39f5f1a44250d5813290a1a8e17879c221eefc8e353a851006bf"}
Oct 30 20:35:28.341075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.341013    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7632ac4b33cb39f5f1a44250d5813290a1a8e17879c221eefc8e353a851006bf"
Oct 30 20:35:28.617861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.617224    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:35:28.617861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.617337    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:35:28.765027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.764984    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:35:28.765212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:28.765054    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:35:29.814605 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:29.814575    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:29.815195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:29.815176    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:29.816420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:29.816396    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:35:29.941517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:29.941491    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:29.941875 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:29.941854    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:29.943159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:29.943136    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:35:31.197769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.197719    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"]
Oct 30 20:35:31.198721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.198698    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8acfdf56-24b9-41eb-b76a-bd3bbea1f323" containerName="revoke-keys"
Oct 30 20:35:31.198806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.198727    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acfdf56-24b9-41eb-b76a-bd3bbea1f323" containerName="revoke-keys"
Oct 30 20:35:31.198919 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.198906    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acfdf56-24b9-41eb-b76a-bd3bbea1f323" containerName="revoke-keys"
Oct 30 20:35:31.202911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.202895    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.258417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.258391    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"]
Oct 30 20:35:31.302752 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.302719    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.302883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.302829    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.302934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.302898    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9p4h\" (UniqueName: \"kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.303847 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.303823    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:31.303942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.303833    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:31.303942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.303891    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:35:31.303942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.303932    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:31.403707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.403681    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9p4h\" (UniqueName: \"kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.403838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.403799    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.403838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.403821    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.404415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.404400    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.406139 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.406113    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.460525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.460461    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9p4h\" (UniqueName: \"kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h\") pod \"maas-api-revoke-sub-0f75dc9b-zbgwq\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.515718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.515682    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:31.888511 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:31.888484    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"]
Oct 30 20:35:31.890158 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:31.890134    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceac69d8_9463_4d48_adb4_27ac3ff88122.slice/crio-ff2012ed315d02f9134df1c2832d12d8a66239a011e2636af5a54bb84a3e422b WatchSource:0}: Error finding container ff2012ed315d02f9134df1c2832d12d8a66239a011e2636af5a54bb84a3e422b: Status 404 returned error can't find the container with id ff2012ed315d02f9134df1c2832d12d8a66239a011e2636af5a54bb84a3e422b
Oct 30 20:35:31.980279 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:35:31.980245    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:35:32.368670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:32.368642    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq" event={"ID":"ceac69d8-9463-4d48-adb4-27ac3ff88122","Type":"ContainerStarted","Data":"a90f8564297c382ee361ae545c4a929c7ad0e59e88172b4843bd06b1803f721f"}
Oct 30 20:35:32.369144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:32.368690    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq" event={"ID":"ceac69d8-9463-4d48-adb4-27ac3ff88122","Type":"ContainerStarted","Data":"ff2012ed315d02f9134df1c2832d12d8a66239a011e2636af5a54bb84a3e422b"}
Oct 30 20:35:32.485608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:32.485558    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq" podStartSLOduration=1.485542448 podStartE2EDuration="1.485542448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:32.483277813 +0000 UTC m=+913.084079773" watchObservedRunningTime="2026-10-30 20:35:32.485542448 +0000 UTC m=+913.086344419"
Oct 30 20:35:33.376555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:33.376518    5967 generic.go:356] "Generic (PLEG): container finished" podID="ceac69d8-9463-4d48-adb4-27ac3ff88122" containerID="a90f8564297c382ee361ae545c4a929c7ad0e59e88172b4843bd06b1803f721f" exitCode=0
Oct 30 20:35:33.377029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:33.376609    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq" event={"ID":"ceac69d8-9463-4d48-adb4-27ac3ff88122","Type":"ContainerDied","Data":"a90f8564297c382ee361ae545c4a929c7ad0e59e88172b4843bd06b1803f721f"}
Oct 30 20:35:34.521703 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.521679    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:34.641693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.641623    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca\") pod \"ceac69d8-9463-4d48-adb4-27ac3ff88122\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") "
Oct 30 20:35:34.641693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.641671    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca\") pod \"ceac69d8-9463-4d48-adb4-27ac3ff88122\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") "
Oct 30 20:35:34.641950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.641770    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h\" (UniqueName: \"kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h\") pod \"ceac69d8-9463-4d48-adb4-27ac3ff88122\" (UID: \"ceac69d8-9463-4d48-adb4-27ac3ff88122\") "
Oct 30 20:35:34.642102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.642078    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca" pod "ceac69d8-9463-4d48-adb4-27ac3ff88122" (UID: "ceac69d8-9463-4d48-adb4-27ac3ff88122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:34.642531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.642505    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:34.643881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.643857    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca" pod "ceac69d8-9463-4d48-adb4-27ac3ff88122" (UID: "ceac69d8-9463-4d48-adb4-27ac3ff88122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:34.644213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.644194    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h" pod "ceac69d8-9463-4d48-adb4-27ac3ff88122" (UID: "ceac69d8-9463-4d48-adb4-27ac3ff88122"). InnerVolumeSpecName "kube-api-access-t9p4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:34.709238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.709200    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"]
Oct 30 20:35:34.709770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.709756    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ceac69d8-9463-4d48-adb4-27ac3ff88122" containerName="revoke-keys"
Oct 30 20:35:34.709823 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.709771    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceac69d8-9463-4d48-adb4-27ac3ff88122" containerName="revoke-keys"
Oct 30 20:35:34.709888 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.709879    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceac69d8-9463-4d48-adb4-27ac3ff88122" containerName="revoke-keys"
Oct 30 20:35:34.713013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.712998    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.744155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.744129    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9p4h\" (UniqueName: \"kubernetes.io/projected/ceac69d8-9463-4d48-adb4-27ac3ff88122-kube-api-access-t9p4h\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:34.744155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.744155    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceac69d8-9463-4d48-adb4-27ac3ff88122-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:34.771335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.771308    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"]
Oct 30 20:35:34.845329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.845302    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46x96\" (UniqueName: \"kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.845446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.845360    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.845536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.845514    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.946993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.946909    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46x96\" (UniqueName: \"kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.947128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.947021    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.947128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.947120    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.947964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.947944    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:34.949565 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:34.949535    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:35.003079 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:35.003053    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46x96\" (UniqueName: \"kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96\") pod \"maas-api-revoke-sub-82bab324-xbvtw\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:35.026171 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:35.026151    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:35.393641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:35.393613    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"
Oct 30 20:35:35.393899 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:35.393645    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq" event={"ID":"ceac69d8-9463-4d48-adb4-27ac3ff88122","Type":"ContainerDied","Data":"ff2012ed315d02f9134df1c2832d12d8a66239a011e2636af5a54bb84a3e422b"}
Oct 30 20:35:35.393899 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:35.393698    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff2012ed315d02f9134df1c2832d12d8a66239a011e2636af5a54bb84a3e422b"
Oct 30 20:35:35.401495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:35.401471    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"]
Oct 30 20:35:35.403528 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:35.403493    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cbffef_dfb1_4388_beb0_60f97c2ebb47.slice/crio-2835d6c3127de1b1a607f6f296333101933744e2a740e2a846ce196836675ea9 WatchSource:0}: Error finding container 2835d6c3127de1b1a607f6f296333101933744e2a740e2a846ce196836675ea9: Status 404 returned error can't find the container with id 2835d6c3127de1b1a607f6f296333101933744e2a740e2a846ce196836675ea9
Oct 30 20:35:36.401477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:36.401442    5967 generic.go:356] "Generic (PLEG): container finished" podID="01cbffef-dfb1-4388-beb0-60f97c2ebb47" containerID="d0bf882a178ea0e14ab10f480399dadc8c2aa0d7ca92b2166d03b1ab90719d46" exitCode=0
Oct 30 20:35:36.401851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:36.401531    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw" event={"ID":"01cbffef-dfb1-4388-beb0-60f97c2ebb47","Type":"ContainerDied","Data":"d0bf882a178ea0e14ab10f480399dadc8c2aa0d7ca92b2166d03b1ab90719d46"}
Oct 30 20:35:36.401851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:36.401583    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw" event={"ID":"01cbffef-dfb1-4388-beb0-60f97c2ebb47","Type":"ContainerStarted","Data":"2835d6c3127de1b1a607f6f296333101933744e2a740e2a846ce196836675ea9"}
Oct 30 20:35:37.558392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.558364    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:37.678425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.678367    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96\" (UniqueName: \"kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96\") pod \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") "
Oct 30 20:35:37.678425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.678411    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca\") pod \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") "
Oct 30 20:35:37.678640 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.678489    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca\") pod \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\" (UID: \"01cbffef-dfb1-4388-beb0-60f97c2ebb47\") "
Oct 30 20:35:37.678884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.678858    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca" pod "01cbffef-dfb1-4388-beb0-60f97c2ebb47" (UID: "01cbffef-dfb1-4388-beb0-60f97c2ebb47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:37.681113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.681087    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96" pod "01cbffef-dfb1-4388-beb0-60f97c2ebb47" (UID: "01cbffef-dfb1-4388-beb0-60f97c2ebb47"). InnerVolumeSpecName "kube-api-access-46x96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:37.682871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.682845    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca" pod "01cbffef-dfb1-4388-beb0-60f97c2ebb47" (UID: "01cbffef-dfb1-4388-beb0-60f97c2ebb47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:37.780106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.780078    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46x96\" (UniqueName: \"kubernetes.io/projected/01cbffef-dfb1-4388-beb0-60f97c2ebb47-kube-api-access-46x96\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:37.780106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.780103    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:37.780255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:37.780112    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cbffef-dfb1-4388-beb0-60f97c2ebb47-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:38.415145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:38.415116    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"
Oct 30 20:35:38.415411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:38.415159    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw" event={"ID":"01cbffef-dfb1-4388-beb0-60f97c2ebb47","Type":"ContainerDied","Data":"2835d6c3127de1b1a607f6f296333101933744e2a740e2a846ce196836675ea9"}
Oct 30 20:35:38.415411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:38.415237    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2835d6c3127de1b1a607f6f296333101933744e2a740e2a846ce196836675ea9"
Oct 30 20:35:40.897925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:40.897893    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"]
Oct 30 20:35:40.898453 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:40.898439    5967 cpu_manager.go:414] "containerMap: removing container" podUID="01cbffef-dfb1-4388-beb0-60f97c2ebb47" containerName="revoke-keys"
Oct 30 20:35:40.898453 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:40.898453    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cbffef-dfb1-4388-beb0-60f97c2ebb47" containerName="revoke-keys"
Oct 30 20:35:40.898565 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:40.898556    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cbffef-dfb1-4388-beb0-60f97c2ebb47" containerName="revoke-keys"
Oct 30 20:35:40.901939 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:40.901921    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:40.954990 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:40.954969    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:41.004270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.004244    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:41.004399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.004304    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:41.004399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.004379    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:35:41.007023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.007007    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"]
Oct 30 20:35:41.012476 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.012453    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx89p\" (UniqueName: \"kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.012581 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.012489    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.012632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.012605    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.113263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.113231    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.113403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.113349    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx89p\" (UniqueName: \"kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.113403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.113381    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.114060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.113992    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.115619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.115598    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.150904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.150850    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"]
Oct 30 20:35:41.154854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.154837    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.169683 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.169661    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx89p\" (UniqueName: \"kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p\") pod \"maas-api-revoke-sub-41880891-kz9x7\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.212587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.212559    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"]
Oct 30 20:35:41.214734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.214719    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:41.314925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.314894    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.315072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.314976    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.315225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.315200    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnx6\" (UniqueName: \"kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.416819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.416698    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.416977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.416874    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnx6\" (UniqueName: \"kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.416977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.416953    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.417420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.417400    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.419377 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.419355    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.474355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.474331    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnx6\" (UniqueName: \"kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6\") pod \"maas-api-revoke-sub-eaaa0c64-9sqrx\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:41.606783 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.606756    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"]
Oct 30 20:35:41.608227 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:41.608188    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41025734_6e53_4794_9ea0_d3498fec8071.slice/crio-a1b9c946f8776aca334fb84cf4088b04878ac0c5720c909da005ed5552ee664d WatchSource:0}: Error finding container a1b9c946f8776aca334fb84cf4088b04878ac0c5720c909da005ed5552ee664d: Status 404 returned error can't find the container with id a1b9c946f8776aca334fb84cf4088b04878ac0c5720c909da005ed5552ee664d
Oct 30 20:35:41.768498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:41.768460    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:42.172762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:42.170128    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"]
Oct 30 20:35:42.453191 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:42.453144    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx" event={"ID":"f58ebaf7-fe37-41d5-974a-4aca6074810f","Type":"ContainerStarted","Data":"8f0fee350aae715c0b3b376c476eb85cff9edb47bd0f877b116342c98934765c"}
Oct 30 20:35:42.455386 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:42.455354    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7" event={"ID":"41025734-6e53-4794-9ea0-d3498fec8071","Type":"ContainerStarted","Data":"154bde4ccd37137132e07804e684f7175007340a12e169f3ab8891ee71eb45b2"}
Oct 30 20:35:42.455545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:42.455411    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7" event={"ID":"41025734-6e53-4794-9ea0-d3498fec8071","Type":"ContainerStarted","Data":"a1b9c946f8776aca334fb84cf4088b04878ac0c5720c909da005ed5552ee664d"}
Oct 30 20:35:42.575288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:42.575177    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7" podStartSLOduration=2.575158276 podStartE2EDuration="2.575158276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:42.574161829 +0000 UTC m=+923.174963789" watchObservedRunningTime="2026-10-30 20:35:42.575158276 +0000 UTC m=+923.175960238"
Oct 30 20:35:43.463664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:43.463630    5967 generic.go:356] "Generic (PLEG): container finished" podID="f58ebaf7-fe37-41d5-974a-4aca6074810f" containerID="5a1a5b3fb0b841b8fa043cde930930e6f20a11e63ea8b73fff322eb58496e250" exitCode=0
Oct 30 20:35:43.464102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:43.463730    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx" event={"ID":"f58ebaf7-fe37-41d5-974a-4aca6074810f","Type":"ContainerDied","Data":"5a1a5b3fb0b841b8fa043cde930930e6f20a11e63ea8b73fff322eb58496e250"}
Oct 30 20:35:43.465463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:43.465433    5967 generic.go:356] "Generic (PLEG): container finished" podID="41025734-6e53-4794-9ea0-d3498fec8071" containerID="154bde4ccd37137132e07804e684f7175007340a12e169f3ab8891ee71eb45b2" exitCode=0
Oct 30 20:35:43.465576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:43.465498    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7" event={"ID":"41025734-6e53-4794-9ea0-d3498fec8071","Type":"ContainerDied","Data":"154bde4ccd37137132e07804e684f7175007340a12e169f3ab8891ee71eb45b2"}
Oct 30 20:35:44.670179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.670151    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:44.674129 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.674109    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:44.770973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.770900    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6\" (UniqueName: \"kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6\") pod \"f58ebaf7-fe37-41d5-974a-4aca6074810f\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") "
Oct 30 20:35:44.770973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.770956    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca\") pod \"f58ebaf7-fe37-41d5-974a-4aca6074810f\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") "
Oct 30 20:35:44.771157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.770996    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca\") pod \"41025734-6e53-4794-9ea0-d3498fec8071\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") "
Oct 30 20:35:44.771191 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771175    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p\" (UniqueName: \"kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p\") pod \"41025734-6e53-4794-9ea0-d3498fec8071\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") "
Oct 30 20:35:44.771230 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771205    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca\") pod \"41025734-6e53-4794-9ea0-d3498fec8071\" (UID: \"41025734-6e53-4794-9ea0-d3498fec8071\") "
Oct 30 20:35:44.771281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771243    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca\") pod \"f58ebaf7-fe37-41d5-974a-4aca6074810f\" (UID: \"f58ebaf7-fe37-41d5-974a-4aca6074810f\") "
Oct 30 20:35:44.771281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771257    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca" pod "f58ebaf7-fe37-41d5-974a-4aca6074810f" (UID: "f58ebaf7-fe37-41d5-974a-4aca6074810f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:44.771550 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771523    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca" pod "41025734-6e53-4794-9ea0-d3498fec8071" (UID: "41025734-6e53-4794-9ea0-d3498fec8071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:44.771780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771734    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:44.771780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.771777    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:44.773253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.773226    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6" pod "f58ebaf7-fe37-41d5-974a-4aca6074810f" (UID: "f58ebaf7-fe37-41d5-974a-4aca6074810f"). InnerVolumeSpecName "kube-api-access-jjnx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:44.773357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.773323    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca" pod "41025734-6e53-4794-9ea0-d3498fec8071" (UID: "41025734-6e53-4794-9ea0-d3498fec8071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:44.773596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.773564    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p" pod "41025734-6e53-4794-9ea0-d3498fec8071" (UID: "41025734-6e53-4794-9ea0-d3498fec8071"). InnerVolumeSpecName "kube-api-access-lx89p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:44.773829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.773807    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca" pod "f58ebaf7-fe37-41d5-974a-4aca6074810f" (UID: "f58ebaf7-fe37-41d5-974a-4aca6074810f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:44.873396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.873363    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnx6\" (UniqueName: \"kubernetes.io/projected/f58ebaf7-fe37-41d5-974a-4aca6074810f-kube-api-access-jjnx6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:44.873396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.873391    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41025734-6e53-4794-9ea0-d3498fec8071-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:44.873574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.873404    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx89p\" (UniqueName: \"kubernetes.io/projected/41025734-6e53-4794-9ea0-d3498fec8071-kube-api-access-lx89p\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:44.873574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:44.873418    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ebaf7-fe37-41d5-974a-4aca6074810f-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:45.480505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:45.480476    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"
Oct 30 20:35:45.480684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:45.480508    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx" event={"ID":"f58ebaf7-fe37-41d5-974a-4aca6074810f","Type":"ContainerDied","Data":"8f0fee350aae715c0b3b376c476eb85cff9edb47bd0f877b116342c98934765c"}
Oct 30 20:35:45.480684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:45.480556    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0fee350aae715c0b3b376c476eb85cff9edb47bd0f877b116342c98934765c"
Oct 30 20:35:45.482265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:45.482238    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7" event={"ID":"41025734-6e53-4794-9ea0-d3498fec8071","Type":"ContainerDied","Data":"a1b9c946f8776aca334fb84cf4088b04878ac0c5720c909da005ed5552ee664d"}
Oct 30 20:35:45.482389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:45.482274    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b9c946f8776aca334fb84cf4088b04878ac0c5720c909da005ed5552ee664d"
Oct 30 20:35:45.482389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:45.482317    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"
Oct 30 20:35:50.882389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.882347    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"]
Oct 30 20:35:50.883202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.883179    5967 cpu_manager.go:414] "containerMap: removing container" podUID="41025734-6e53-4794-9ea0-d3498fec8071" containerName="revoke-keys"
Oct 30 20:35:50.883202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.883204    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41025734-6e53-4794-9ea0-d3498fec8071" containerName="revoke-keys"
Oct 30 20:35:50.883357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.883238    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f58ebaf7-fe37-41d5-974a-4aca6074810f" containerName="revoke-keys"
Oct 30 20:35:50.883357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.883246    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58ebaf7-fe37-41d5-974a-4aca6074810f" containerName="revoke-keys"
Oct 30 20:35:50.883456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.883396    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="41025734-6e53-4794-9ea0-d3498fec8071" containerName="revoke-keys"
Oct 30 20:35:50.883456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.883413    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58ebaf7-fe37-41d5-974a-4aca6074810f" containerName="revoke-keys"
Oct 30 20:35:50.887101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.887081    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:50.940552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.940518    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpshx\" (UniqueName: \"kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:50.940702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.940663    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:50.940802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.940730    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:50.945209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.945184    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"]
Oct 30 20:35:50.950476 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.950450    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:50.950603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.950508    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:35:50.990828 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.990811    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:50.991005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:50.990988    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:51.042113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.042085    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpshx\" (UniqueName: \"kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.042271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.042173    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.042271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.042193    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.042855 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.042833    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.044689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.044672    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.099815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.099784    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpshx\" (UniqueName: \"kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx\") pod \"maas-api-revoke-sub-000fe0f5-5lgvs\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.199722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.199661    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:51.572461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:51.572433    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"]
Oct 30 20:35:51.573860 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:51.573833    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0430ee17_c021_4a92_8835_93d40ad2db2a.slice/crio-e34c5016c5be1090a8f4297acdc93b6e8600dc1d84a85c261dca3b1d06c65892 WatchSource:0}: Error finding container e34c5016c5be1090a8f4297acdc93b6e8600dc1d84a85c261dca3b1d06c65892: Status 404 returned error can't find the container with id e34c5016c5be1090a8f4297acdc93b6e8600dc1d84a85c261dca3b1d06c65892
Oct 30 20:35:52.525265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:52.525228    5967 generic.go:356] "Generic (PLEG): container finished" podID="0430ee17-c021-4a92-8835-93d40ad2db2a" containerID="776bac5333a5cb24dda9e6ed3c2e69bf66b27d428b18feb958eb06a8042ac23d" exitCode=0
Oct 30 20:35:52.525654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:52.525319    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs" event={"ID":"0430ee17-c021-4a92-8835-93d40ad2db2a","Type":"ContainerDied","Data":"776bac5333a5cb24dda9e6ed3c2e69bf66b27d428b18feb958eb06a8042ac23d"}
Oct 30 20:35:52.525654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:52.525370    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs" event={"ID":"0430ee17-c021-4a92-8835-93d40ad2db2a","Type":"ContainerStarted","Data":"e34c5016c5be1090a8f4297acdc93b6e8600dc1d84a85c261dca3b1d06c65892"}
Oct 30 20:35:53.685375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.685353    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:53.771452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.771428    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca\") pod \"0430ee17-c021-4a92-8835-93d40ad2db2a\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") "
Oct 30 20:35:53.771592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.771543    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca\") pod \"0430ee17-c021-4a92-8835-93d40ad2db2a\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") "
Oct 30 20:35:53.771654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.771622    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx\" (UniqueName: \"kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx\") pod \"0430ee17-c021-4a92-8835-93d40ad2db2a\" (UID: \"0430ee17-c021-4a92-8835-93d40ad2db2a\") "
Oct 30 20:35:53.771754 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.771719    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca" pod "0430ee17-c021-4a92-8835-93d40ad2db2a" (UID: "0430ee17-c021-4a92-8835-93d40ad2db2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:53.771894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.771876    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:53.773682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.773664    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca" pod "0430ee17-c021-4a92-8835-93d40ad2db2a" (UID: "0430ee17-c021-4a92-8835-93d40ad2db2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:53.773815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.773694    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx" pod "0430ee17-c021-4a92-8835-93d40ad2db2a" (UID: "0430ee17-c021-4a92-8835-93d40ad2db2a"). InnerVolumeSpecName "kube-api-access-mpshx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:53.873288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.873263    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430ee17-c021-4a92-8835-93d40ad2db2a-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:53.873288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:53.873285    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpshx\" (UniqueName: \"kubernetes.io/projected/0430ee17-c021-4a92-8835-93d40ad2db2a-kube-api-access-mpshx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:54.538532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.538505    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"
Oct 30 20:35:54.538699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.538544    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs" event={"ID":"0430ee17-c021-4a92-8835-93d40ad2db2a","Type":"ContainerDied","Data":"e34c5016c5be1090a8f4297acdc93b6e8600dc1d84a85c261dca3b1d06c65892"}
Oct 30 20:35:54.538699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.538595    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34c5016c5be1090a8f4297acdc93b6e8600dc1d84a85c261dca3b1d06c65892"
Oct 30 20:35:54.965061 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.965025    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"]
Oct 30 20:35:54.965692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.965672    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0430ee17-c021-4a92-8835-93d40ad2db2a" containerName="revoke-keys"
Oct 30 20:35:54.965692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.965691    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0430ee17-c021-4a92-8835-93d40ad2db2a" containerName="revoke-keys"
Oct 30 20:35:54.965861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.965805    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0430ee17-c021-4a92-8835-93d40ad2db2a" containerName="revoke-keys"
Oct 30 20:35:54.969203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:54.969183    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.022419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.022391    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:55.022533 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.022409    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:35:55.023297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.023279    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:55.026144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.026124    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"]
Oct 30 20:35:55.071659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.071635    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:55.087245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.087213    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2q45\" (UniqueName: \"kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.087245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.087249    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.087454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.087282    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.188292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.188255    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2q45\" (UniqueName: \"kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.188475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.188299    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.188475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.188343    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.188977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.188957    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.190878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.190863    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.244869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.244812    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2q45\" (UniqueName: \"kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45\") pod \"maas-api-revoke-sub-1886feb4-j9kxj\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.282525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.282502    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:55.659840 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:55.659814    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"]
Oct 30 20:35:55.661220 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:55.661192    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8e6ff5_56ee_401a_9354_89486b6822bd.slice/crio-f3ea181c065e0513d95bbdcdf68a307b9948cfbe93464fe00e35eb2fff5822e5 WatchSource:0}: Error finding container f3ea181c065e0513d95bbdcdf68a307b9948cfbe93464fe00e35eb2fff5822e5: Status 404 returned error can't find the container with id f3ea181c065e0513d95bbdcdf68a307b9948cfbe93464fe00e35eb2fff5822e5
Oct 30 20:35:56.552371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.552347    5967 generic.go:356] "Generic (PLEG): container finished" podID="fe8e6ff5-56ee-401a-9354-89486b6822bd" containerID="67df9b787646a8b2fc4ed3cd522d33d534758b60bbc9212e57d4e6051684c209" exitCode=0
Oct 30 20:35:56.552727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.552433    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj" event={"ID":"fe8e6ff5-56ee-401a-9354-89486b6822bd","Type":"ContainerDied","Data":"67df9b787646a8b2fc4ed3cd522d33d534758b60bbc9212e57d4e6051684c209"}
Oct 30 20:35:56.552727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.552496    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj" event={"ID":"fe8e6ff5-56ee-401a-9354-89486b6822bd","Type":"ContainerStarted","Data":"f3ea181c065e0513d95bbdcdf68a307b9948cfbe93464fe00e35eb2fff5822e5"}
Oct 30 20:35:56.598837 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.598813    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"]
Oct 30 20:35:56.602630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.602613    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.659635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.659609    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"]
Oct 30 20:35:56.704676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.704620    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.704676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.704657    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzp9c\" (UniqueName: \"kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.704879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.704687    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.805699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.805630    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.805858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.805790    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.805858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.805819    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp9c\" (UniqueName: \"kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.806389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.806358    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.808285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.808260    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.863039 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.863012    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp9c\" (UniqueName: \"kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c\") pod \"maas-api-revoke-sub-318093ab-j4r2r\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:56.949079 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:56.949056    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:57.324696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.324667    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"]
Oct 30 20:35:57.325565 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:57.325539    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f7835c_ee10_4439_82ef_9554af6ec202.slice/crio-39d1af7d50d37c4301383cd7762ccab17339bb9fbe0a750323c89e6ddb5e668d WatchSource:0}: Error finding container 39d1af7d50d37c4301383cd7762ccab17339bb9fbe0a750323c89e6ddb5e668d: Status 404 returned error can't find the container with id 39d1af7d50d37c4301383cd7762ccab17339bb9fbe0a750323c89e6ddb5e668d
Oct 30 20:35:57.560353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.560319    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r" event={"ID":"f3f7835c-ee10-4439-82ef-9554af6ec202","Type":"ContainerStarted","Data":"39d1af7d50d37c4301383cd7762ccab17339bb9fbe0a750323c89e6ddb5e668d"}
Oct 30 20:35:57.696622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.696601    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:57.817222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.817196    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45\" (UniqueName: \"kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45\") pod \"fe8e6ff5-56ee-401a-9354-89486b6822bd\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") "
Oct 30 20:35:57.817405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.817274    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca\") pod \"fe8e6ff5-56ee-401a-9354-89486b6822bd\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") "
Oct 30 20:35:57.817405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.817380    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca\") pod \"fe8e6ff5-56ee-401a-9354-89486b6822bd\" (UID: \"fe8e6ff5-56ee-401a-9354-89486b6822bd\") "
Oct 30 20:35:57.817775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.817726    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca" pod "fe8e6ff5-56ee-401a-9354-89486b6822bd" (UID: "fe8e6ff5-56ee-401a-9354-89486b6822bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:57.817902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.817861    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:57.819395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.819368    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45" pod "fe8e6ff5-56ee-401a-9354-89486b6822bd" (UID: "fe8e6ff5-56ee-401a-9354-89486b6822bd"). InnerVolumeSpecName "kube-api-access-l2q45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:57.819663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.819643    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca" pod "fe8e6ff5-56ee-401a-9354-89486b6822bd" (UID: "fe8e6ff5-56ee-401a-9354-89486b6822bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:57.919144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.919077    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2q45\" (UniqueName: \"kubernetes.io/projected/fe8e6ff5-56ee-401a-9354-89486b6822bd-kube-api-access-l2q45\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:57.919144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:57.919102    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe8e6ff5-56ee-401a-9354-89486b6822bd-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:58.567888 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.567860    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"
Oct 30 20:35:58.568262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.567861    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj" event={"ID":"fe8e6ff5-56ee-401a-9354-89486b6822bd","Type":"ContainerDied","Data":"f3ea181c065e0513d95bbdcdf68a307b9948cfbe93464fe00e35eb2fff5822e5"}
Oct 30 20:35:58.568262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.567991    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ea181c065e0513d95bbdcdf68a307b9948cfbe93464fe00e35eb2fff5822e5"
Oct 30 20:35:58.569525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.569488    5967 generic.go:356] "Generic (PLEG): container finished" podID="f3f7835c-ee10-4439-82ef-9554af6ec202" containerID="8365a804326a9502576560b91ea607c2509fcc1d6676f222fc8e5229bccc9f75" exitCode=0
Oct 30 20:35:58.569660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.569553    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r" event={"ID":"f3f7835c-ee10-4439-82ef-9554af6ec202","Type":"ContainerDied","Data":"8365a804326a9502576560b91ea607c2509fcc1d6676f222fc8e5229bccc9f75"}
Oct 30 20:35:58.861276 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.861205    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"]
Oct 30 20:35:58.861858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.861844    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fe8e6ff5-56ee-401a-9354-89486b6822bd" containerName="revoke-keys"
Oct 30 20:35:58.861897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.861859    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8e6ff5-56ee-401a-9354-89486b6822bd" containerName="revoke-keys"
Oct 30 20:35:58.861956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.861947    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8e6ff5-56ee-401a-9354-89486b6822bd" containerName="revoke-keys"
Oct 30 20:35:58.865361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.865341    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:58.921663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.921638    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"]
Oct 30 20:35:58.929592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.929569    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:58.929721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.929698    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:58.929842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:58.929808    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpb2d\" (UniqueName: \"kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.030688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.030665    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.030835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.030775    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.030890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.030838    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpb2d\" (UniqueName: \"kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.031432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.031414    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.033160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.033138    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.087373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.087351    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpb2d\" (UniqueName: \"kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d\") pod \"maas-api-revoke-sub-fee8d684-lwmsn\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.112522 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.112466    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"]
Oct 30 20:35:59.116751 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.116728    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.172978 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.172951    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"]
Oct 30 20:35:59.178680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.178653    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:35:59.233578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.233041    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.233578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.233214    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.233578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.233248    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kb6\" (UniqueName: \"kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.334980 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.334950    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.335163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.335028    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.335163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.335044    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kb6\" (UniqueName: \"kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.335711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.335688    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.337299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.337282    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.391507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.391453    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kb6\" (UniqueName: \"kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6\") pod \"maas-api-revoke-sub-0409d895-7dwzj\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.429905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.429886    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:35:59.558716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.558695    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"]
Oct 30 20:35:59.560988 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:59.560949    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bba0197_e717_42b1_8a57_0594d43b0f38.slice/crio-b346fbd4923c2fbff91617abcae1dc9ee8059138851e83842a41520a83841b0f WatchSource:0}: Error finding container b346fbd4923c2fbff91617abcae1dc9ee8059138851e83842a41520a83841b0f: Status 404 returned error can't find the container with id b346fbd4923c2fbff91617abcae1dc9ee8059138851e83842a41520a83841b0f
Oct 30 20:35:59.577654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.577633    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn" event={"ID":"6bba0197-e717-42b1-8a57-0594d43b0f38","Type":"ContainerStarted","Data":"b346fbd4923c2fbff91617abcae1dc9ee8059138851e83842a41520a83841b0f"}
Oct 30 20:35:59.691638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.691615    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:35:59.739355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.739331    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca\") pod \"f3f7835c-ee10-4439-82ef-9554af6ec202\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") "
Oct 30 20:35:59.739461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.739370    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca\") pod \"f3f7835c-ee10-4439-82ef-9554af6ec202\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") "
Oct 30 20:35:59.739461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.739394    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c\" (UniqueName: \"kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c\") pod \"f3f7835c-ee10-4439-82ef-9554af6ec202\" (UID: \"f3f7835c-ee10-4439-82ef-9554af6ec202\") "
Oct 30 20:35:59.739664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.739644    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca" pod "f3f7835c-ee10-4439-82ef-9554af6ec202" (UID: "f3f7835c-ee10-4439-82ef-9554af6ec202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:59.740014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.739951    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:59.741358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.741338    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca" pod "f3f7835c-ee10-4439-82ef-9554af6ec202" (UID: "f3f7835c-ee10-4439-82ef-9554af6ec202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:59.741440 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.741418    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c" pod "f3f7835c-ee10-4439-82ef-9554af6ec202" (UID: "f3f7835c-ee10-4439-82ef-9554af6ec202"). InnerVolumeSpecName "kube-api-access-bzp9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:59.806759 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.806719    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"]
Oct 30 20:35:59.808154 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:35:59.808128    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47798542_aa0e_48bf_a15f_b25de925551a.slice/crio-6bdeb38019b6719b6c12139e8ae89fe1ac0d9c2b013fee085c37e48590fa05d0 WatchSource:0}: Error finding container 6bdeb38019b6719b6c12139e8ae89fe1ac0d9c2b013fee085c37e48590fa05d0: Status 404 returned error can't find the container with id 6bdeb38019b6719b6c12139e8ae89fe1ac0d9c2b013fee085c37e48590fa05d0
Oct 30 20:35:59.841265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.841239    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7835c-ee10-4439-82ef-9554af6ec202-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:59.841265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:35:59.841266    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzp9c\" (UniqueName: \"kubernetes.io/projected/f3f7835c-ee10-4439-82ef-9554af6ec202-kube-api-access-bzp9c\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:00.585175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.585143    5967 generic.go:356] "Generic (PLEG): container finished" podID="6bba0197-e717-42b1-8a57-0594d43b0f38" containerID="0f6f552299075e20ea7dca66ff40873bf2d0565a6fb6c6923411d92efdc327de" exitCode=0
Oct 30 20:36:00.585613 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.585242    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn" event={"ID":"6bba0197-e717-42b1-8a57-0594d43b0f38","Type":"ContainerDied","Data":"0f6f552299075e20ea7dca66ff40873bf2d0565a6fb6c6923411d92efdc327de"}
Oct 30 20:36:00.587151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.587127    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj" event={"ID":"47798542-aa0e-48bf-a15f-b25de925551a","Type":"ContainerStarted","Data":"831e2d1b9f257db8b6f0193668cc2306644c545ff1a0c2b3be0c6f9f836db42b"}
Oct 30 20:36:00.587257 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.587167    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj" event={"ID":"47798542-aa0e-48bf-a15f-b25de925551a","Type":"ContainerStarted","Data":"6bdeb38019b6719b6c12139e8ae89fe1ac0d9c2b013fee085c37e48590fa05d0"}
Oct 30 20:36:00.588915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.588896    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r" event={"ID":"f3f7835c-ee10-4439-82ef-9554af6ec202","Type":"ContainerDied","Data":"39d1af7d50d37c4301383cd7762ccab17339bb9fbe0a750323c89e6ddb5e668d"}
Oct 30 20:36:00.589030 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.588931    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d1af7d50d37c4301383cd7762ccab17339bb9fbe0a750323c89e6ddb5e668d"
Oct 30 20:36:00.589030 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.589012    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"
Oct 30 20:36:00.811874 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:00.811823    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj" podStartSLOduration=1.8118094660000001 podStartE2EDuration="1.811809466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:00.810491463 +0000 UTC m=+941.411293425" watchObservedRunningTime="2026-10-30 20:36:00.811809466 +0000 UTC m=+941.412611465"
Oct 30 20:36:01.596563 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.596532    5967 generic.go:356] "Generic (PLEG): container finished" podID="47798542-aa0e-48bf-a15f-b25de925551a" containerID="831e2d1b9f257db8b6f0193668cc2306644c545ff1a0c2b3be0c6f9f836db42b" exitCode=0
Oct 30 20:36:01.597057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.596608    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj" event={"ID":"47798542-aa0e-48bf-a15f-b25de925551a","Type":"ContainerDied","Data":"831e2d1b9f257db8b6f0193668cc2306644c545ff1a0c2b3be0c6f9f836db42b"}
Oct 30 20:36:01.747698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.747675    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:36:01.864237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.864164    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca\") pod \"6bba0197-e717-42b1-8a57-0594d43b0f38\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") "
Oct 30 20:36:01.864577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.864555    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca\") pod \"6bba0197-e717-42b1-8a57-0594d43b0f38\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") "
Oct 30 20:36:01.864694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.864660    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca" pod "6bba0197-e717-42b1-8a57-0594d43b0f38" (UID: "6bba0197-e717-42b1-8a57-0594d43b0f38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:01.864694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.864668    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d\" (UniqueName: \"kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d\") pod \"6bba0197-e717-42b1-8a57-0594d43b0f38\" (UID: \"6bba0197-e717-42b1-8a57-0594d43b0f38\") "
Oct 30 20:36:01.867769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.865270    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:01.867896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.867796    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca" pod "6bba0197-e717-42b1-8a57-0594d43b0f38" (UID: "6bba0197-e717-42b1-8a57-0594d43b0f38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:01.868213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.868191    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d" pod "6bba0197-e717-42b1-8a57-0594d43b0f38" (UID: "6bba0197-e717-42b1-8a57-0594d43b0f38"). InnerVolumeSpecName "kube-api-access-jpb2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:01.966004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.965978    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0197-e717-42b1-8a57-0594d43b0f38-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:01.966004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:01.966002    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpb2d\" (UniqueName: \"kubernetes.io/projected/6bba0197-e717-42b1-8a57-0594d43b0f38-kube-api-access-jpb2d\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:01.979189 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:36:01.979160    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:36:02.604712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.604683    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"
Oct 30 20:36:02.605207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.604712    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn" event={"ID":"6bba0197-e717-42b1-8a57-0594d43b0f38","Type":"ContainerDied","Data":"b346fbd4923c2fbff91617abcae1dc9ee8059138851e83842a41520a83841b0f"}
Oct 30 20:36:02.605207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.604799    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b346fbd4923c2fbff91617abcae1dc9ee8059138851e83842a41520a83841b0f"
Oct 30 20:36:02.744078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.744054    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:36:02.877155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.877073    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca\") pod \"47798542-aa0e-48bf-a15f-b25de925551a\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") "
Oct 30 20:36:02.877155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.877126    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca\") pod \"47798542-aa0e-48bf-a15f-b25de925551a\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") "
Oct 30 20:36:02.877400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.877166    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6\" (UniqueName: \"kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6\") pod \"47798542-aa0e-48bf-a15f-b25de925551a\" (UID: \"47798542-aa0e-48bf-a15f-b25de925551a\") "
Oct 30 20:36:02.877858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.877525    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca" pod "47798542-aa0e-48bf-a15f-b25de925551a" (UID: "47798542-aa0e-48bf-a15f-b25de925551a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:02.877858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.877716    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:02.879710 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.879690    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6" pod "47798542-aa0e-48bf-a15f-b25de925551a" (UID: "47798542-aa0e-48bf-a15f-b25de925551a"). InnerVolumeSpecName "kube-api-access-c4kb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:02.880829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.880804    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca" pod "47798542-aa0e-48bf-a15f-b25de925551a" (UID: "47798542-aa0e-48bf-a15f-b25de925551a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:02.979594 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.979555    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47798542-aa0e-48bf-a15f-b25de925551a-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:02.979594 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:02.979591    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4kb6\" (UniqueName: \"kubernetes.io/projected/47798542-aa0e-48bf-a15f-b25de925551a-kube-api-access-c4kb6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:03.620595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:03.617860    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj" event={"ID":"47798542-aa0e-48bf-a15f-b25de925551a","Type":"ContainerDied","Data":"6bdeb38019b6719b6c12139e8ae89fe1ac0d9c2b013fee085c37e48590fa05d0"}
Oct 30 20:36:03.620595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:03.617924    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bdeb38019b6719b6c12139e8ae89fe1ac0d9c2b013fee085c37e48590fa05d0"
Oct 30 20:36:03.620595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:03.617993    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"
Oct 30 20:36:05.767069 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767024    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"]
Oct 30 20:36:05.767653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767637    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f3f7835c-ee10-4439-82ef-9554af6ec202" containerName="revoke-keys"
Oct 30 20:36:05.767698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767656    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f7835c-ee10-4439-82ef-9554af6ec202" containerName="revoke-keys"
Oct 30 20:36:05.767698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767683    5967 cpu_manager.go:414] "containerMap: removing container" podUID="47798542-aa0e-48bf-a15f-b25de925551a" containerName="revoke-keys"
Oct 30 20:36:05.767698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767688    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47798542-aa0e-48bf-a15f-b25de925551a" containerName="revoke-keys"
Oct 30 20:36:05.767813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767709    5967 cpu_manager.go:414] "containerMap: removing container" podUID="6bba0197-e717-42b1-8a57-0594d43b0f38" containerName="revoke-keys"
Oct 30 20:36:05.767813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767714    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba0197-e717-42b1-8a57-0594d43b0f38" containerName="revoke-keys"
Oct 30 20:36:05.767881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767814    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="47798542-aa0e-48bf-a15f-b25de925551a" containerName="revoke-keys"
Oct 30 20:36:05.767881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767832    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba0197-e717-42b1-8a57-0594d43b0f38" containerName="revoke-keys"
Oct 30 20:36:05.767881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.767845    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f7835c-ee10-4439-82ef-9554af6ec202" containerName="revoke-keys"
Oct 30 20:36:05.771173 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.771155    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.826152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.826118    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:05.846722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.846685    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.846885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.846777    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.846990 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.846964    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjmj\" (UniqueName: \"kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.874954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.874919    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:05.875497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.875479    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:05.875598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.875553    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:36:05.878068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.878039    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"]
Oct 30 20:36:05.948513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.948478    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.948731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.948540    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.948731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.948652    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjmj\" (UniqueName: \"kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.949291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.949266    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:05.951630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:05.951603    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:06.016915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.016883    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjmj\" (UniqueName: \"kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj\") pod \"maas-api-revoke-sub-ee5f4e4f-prbfv\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:06.084540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.084453    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:06.461792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.461767    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"]
Oct 30 20:36:06.463288 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:06.463248    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff37fd42_38e1_4d90_a833_cce6ae42b829.slice/crio-f5fc2cf47fa144c8a83e280338e33de48f35b0516ba2289e6def05144ce6c5ef WatchSource:0}: Error finding container f5fc2cf47fa144c8a83e280338e33de48f35b0516ba2289e6def05144ce6c5ef: Status 404 returned error can't find the container with id f5fc2cf47fa144c8a83e280338e33de48f35b0516ba2289e6def05144ce6c5ef
Oct 30 20:36:06.639117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.639082    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv" event={"ID":"ff37fd42-38e1-4d90-a833-cce6ae42b829","Type":"ContainerStarted","Data":"f5fc2cf47fa144c8a83e280338e33de48f35b0516ba2289e6def05144ce6c5ef"}
Oct 30 20:36:06.927981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.927949    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"]
Oct 30 20:36:06.931985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.931967    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:06.989970 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:06.989932    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"]
Oct 30 20:36:07.064077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.063989    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.064077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.064043    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.064256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.064155    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzct8\" (UniqueName: \"kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.165918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.165885    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.166148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.165934    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.166148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.165968    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzct8\" (UniqueName: \"kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.166576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.166554    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.168441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.168420    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.224826 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.224796    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzct8\" (UniqueName: \"kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8\") pod \"maas-api-revoke-sub-db988f65-djch6\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.245052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.245012    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:07.626629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.626592    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"]
Oct 30 20:36:07.627892 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:07.627865    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecea26b_71cf_46c5_84e9_3bc3ee7f5dff.slice/crio-9f19c5a8eab764618bed3e953414719acaa067a9246953b299af7b75a105a203 WatchSource:0}: Error finding container 9f19c5a8eab764618bed3e953414719acaa067a9246953b299af7b75a105a203: Status 404 returned error can't find the container with id 9f19c5a8eab764618bed3e953414719acaa067a9246953b299af7b75a105a203
Oct 30 20:36:07.649989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.649941    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6" event={"ID":"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff","Type":"ContainerStarted","Data":"9f19c5a8eab764618bed3e953414719acaa067a9246953b299af7b75a105a203"}
Oct 30 20:36:07.651721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.651690    5967 generic.go:356] "Generic (PLEG): container finished" podID="ff37fd42-38e1-4d90-a833-cce6ae42b829" containerID="f14153840d845c46994a786c0f584fdb94dda44076d69dd9e603b222bd7b49e3" exitCode=0
Oct 30 20:36:07.651924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:07.651759    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv" event={"ID":"ff37fd42-38e1-4d90-a833-cce6ae42b829","Type":"ContainerDied","Data":"f14153840d845c46994a786c0f584fdb94dda44076d69dd9e603b222bd7b49e3"}
Oct 30 20:36:08.659833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.659795    5967 generic.go:356] "Generic (PLEG): container finished" podID="3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" containerID="194000d985a2cfd9798c18af3246d14debf1ae5b18bc0aa2e392565506ce0d19" exitCode=0
Oct 30 20:36:08.660212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.659890    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6" event={"ID":"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff","Type":"ContainerDied","Data":"194000d985a2cfd9798c18af3246d14debf1ae5b18bc0aa2e392565506ce0d19"}
Oct 30 20:36:08.813152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.813117    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:08.986492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.986458    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca\") pod \"ff37fd42-38e1-4d90-a833-cce6ae42b829\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") "
Oct 30 20:36:08.986675 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.986567    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca\") pod \"ff37fd42-38e1-4d90-a833-cce6ae42b829\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") "
Oct 30 20:36:08.986675 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.986613    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj\" (UniqueName: \"kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj\") pod \"ff37fd42-38e1-4d90-a833-cce6ae42b829\" (UID: \"ff37fd42-38e1-4d90-a833-cce6ae42b829\") "
Oct 30 20:36:08.987015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.986989    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca" pod "ff37fd42-38e1-4d90-a833-cce6ae42b829" (UID: "ff37fd42-38e1-4d90-a833-cce6ae42b829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:08.987108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.987085    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:08.988840 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.988816    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca" pod "ff37fd42-38e1-4d90-a833-cce6ae42b829" (UID: "ff37fd42-38e1-4d90-a833-cce6ae42b829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:08.988907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:08.988851    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj" pod "ff37fd42-38e1-4d90-a833-cce6ae42b829" (UID: "ff37fd42-38e1-4d90-a833-cce6ae42b829"). InnerVolumeSpecName "kube-api-access-pkjmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:09.088765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:09.088661    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37fd42-38e1-4d90-a833-cce6ae42b829-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:09.088765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:09.088694    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjmj\" (UniqueName: \"kubernetes.io/projected/ff37fd42-38e1-4d90-a833-cce6ae42b829-kube-api-access-pkjmj\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:09.667358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:09.667321    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"
Oct 30 20:36:09.667358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:09.667348    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv" event={"ID":"ff37fd42-38e1-4d90-a833-cce6ae42b829","Type":"ContainerDied","Data":"f5fc2cf47fa144c8a83e280338e33de48f35b0516ba2289e6def05144ce6c5ef"}
Oct 30 20:36:09.667932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:09.667393    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5fc2cf47fa144c8a83e280338e33de48f35b0516ba2289e6def05144ce6c5ef"
Oct 30 20:36:09.819425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:09.819355    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:10.000544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.000509    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8\" (UniqueName: \"kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8\") pod \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") "
Oct 30 20:36:10.000724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.000624    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca\") pod \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") "
Oct 30 20:36:10.000724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.000655    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca\") pod \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\" (UID: \"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff\") "
Oct 30 20:36:10.001047 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.001023    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca" pod "3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" (UID: "3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:10.002965 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.002940    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca" pod "3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" (UID: "3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:10.003121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.003101    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8" pod "3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" (UID: "3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff"). InnerVolumeSpecName "kube-api-access-mzct8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:10.102137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.102042    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzct8\" (UniqueName: \"kubernetes.io/projected/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-kube-api-access-mzct8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.102137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.102076    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.102137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.102088    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.675399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.675369    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"
Oct 30 20:36:10.675838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.675412    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6" event={"ID":"3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff","Type":"ContainerDied","Data":"9f19c5a8eab764618bed3e953414719acaa067a9246953b299af7b75a105a203"}
Oct 30 20:36:10.675838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:10.675460    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f19c5a8eab764618bed3e953414719acaa067a9246953b299af7b75a105a203"
Oct 30 20:36:15.234698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.234662    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"]
Oct 30 20:36:15.235260 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.235245    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" containerName="revoke-keys"
Oct 30 20:36:15.235299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.235261    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" containerName="revoke-keys"
Oct 30 20:36:15.235299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.235289    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ff37fd42-38e1-4d90-a833-cce6ae42b829" containerName="revoke-keys"
Oct 30 20:36:15.235299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.235294    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff37fd42-38e1-4d90-a833-cce6ae42b829" containerName="revoke-keys"
Oct 30 20:36:15.235405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.235368    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" containerName="revoke-keys"
Oct 30 20:36:15.235405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.235380    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff37fd42-38e1-4d90-a833-cce6ae42b829" containerName="revoke-keys"
Oct 30 20:36:15.238959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.238938    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.262164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.262130    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.262336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.262246    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.262336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.262324    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.341115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.340993    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:15.341115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.341003    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:36:15.341115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.341054    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:15.341397 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.341184    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:15.344237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.344214    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"]
Oct 30 20:36:15.363981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.363943    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.364174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.364031    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.364174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.364081    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.364729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.364708    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.366503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.366484    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.421094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.421060    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln\") pod \"maas-api-revoke-sub-e56a95ac-c2nws\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.552682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.552592    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:15.935579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:15.935535    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"]
Oct 30 20:36:16.718005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:16.717969    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws" event={"ID":"c0cef49d-08cd-4424-9c71-fb6bb8561e26","Type":"ContainerStarted","Data":"cdd5e01aeba109f32b98830d4be2b4ae6fb617add709e462f4e45d465e47c2df"}
Oct 30 20:36:16.718430 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:16.718024    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws" event={"ID":"c0cef49d-08cd-4424-9c71-fb6bb8561e26","Type":"ContainerStarted","Data":"03e6958e2a34556f62435f82dad6cf48f440991ec6f2980134e2cb94fecbed73"}
Oct 30 20:36:16.834931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:16.834877    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws" podStartSLOduration=1.834861176 podStartE2EDuration="1.834861176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:16.83149777 +0000 UTC m=+957.432299729" watchObservedRunningTime="2026-10-30 20:36:16.834861176 +0000 UTC m=+957.435663158"
Oct 30 20:36:17.726717 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:17.726686    5967 generic.go:356] "Generic (PLEG): container finished" podID="c0cef49d-08cd-4424-9c71-fb6bb8561e26" containerID="cdd5e01aeba109f32b98830d4be2b4ae6fb617add709e462f4e45d465e47c2df" exitCode=0
Oct 30 20:36:17.727149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:17.726798    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws" event={"ID":"c0cef49d-08cd-4424-9c71-fb6bb8561e26","Type":"ContainerDied","Data":"cdd5e01aeba109f32b98830d4be2b4ae6fb617add709e462f4e45d465e47c2df"}
Oct 30 20:36:18.889390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:18.889363    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:19.008230 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.008135    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca\") pod \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") "
Oct 30 20:36:19.008643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.008618    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca\") pod \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") "
Oct 30 20:36:19.008934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.008908    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln\") pod \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\" (UID: \"c0cef49d-08cd-4424-9c71-fb6bb8561e26\") "
Oct 30 20:36:19.009242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.009211    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca" pod "c0cef49d-08cd-4424-9c71-fb6bb8561e26" (UID: "c0cef49d-08cd-4424-9c71-fb6bb8561e26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:19.009651 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.009632    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:19.014760 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.012297    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca" pod "c0cef49d-08cd-4424-9c71-fb6bb8561e26" (UID: "c0cef49d-08cd-4424-9c71-fb6bb8561e26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:19.025803 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.015588    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln" pod "c0cef49d-08cd-4424-9c71-fb6bb8561e26" (UID: "c0cef49d-08cd-4424-9c71-fb6bb8561e26"). InnerVolumeSpecName "kube-api-access-rzvln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:19.110033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.109994    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"]
Oct 30 20:36:19.110555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.110533    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0cef49d-08cd-4424-9c71-fb6bb8561e26-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:19.110594 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.110560    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/c0cef49d-08cd-4424-9c71-fb6bb8561e26-kube-api-access-rzvln\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:19.110886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.110869    5967 cpu_manager.go:414] "containerMap: removing container" podUID="c0cef49d-08cd-4424-9c71-fb6bb8561e26" containerName="revoke-keys"
Oct 30 20:36:19.110922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.110890    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cef49d-08cd-4424-9c71-fb6bb8561e26" containerName="revoke-keys"
Oct 30 20:36:19.111115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.111102    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cef49d-08cd-4424-9c71-fb6bb8561e26" containerName="revoke-keys"
Oct 30 20:36:19.115221 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.115199    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.169188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.169155    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"]
Oct 30 20:36:19.211716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.211675    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.211909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.211890    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.211948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.211931    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh47p\" (UniqueName: \"kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.313278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.313188    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.313278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.313261    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh47p\" (UniqueName: \"kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.313513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.313352    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.313940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.313914    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.315979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.315958    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.370564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.370529    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh47p\" (UniqueName: \"kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p\") pod \"maas-api-revoke-sub-bf614417-65jzv\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.433566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.433532    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:19.771766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.752308    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws" event={"ID":"c0cef49d-08cd-4424-9c71-fb6bb8561e26","Type":"ContainerDied","Data":"03e6958e2a34556f62435f82dad6cf48f440991ec6f2980134e2cb94fecbed73"}
Oct 30 20:36:19.771766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.752365    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03e6958e2a34556f62435f82dad6cf48f440991ec6f2980134e2cb94fecbed73"
Oct 30 20:36:19.771766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.752437    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"
Oct 30 20:36:19.816059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:19.816030    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"]
Oct 30 20:36:19.818348 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:19.818313    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf1259a_192e_4271_9bdd_d602cbc7a47e.slice/crio-f9b27c3f9db54cdd529e53cd4e4b52cbbd019de41bd909e20cca27677ea1732f WatchSource:0}: Error finding container f9b27c3f9db54cdd529e53cd4e4b52cbbd019de41bd909e20cca27677ea1732f: Status 404 returned error can't find the container with id f9b27c3f9db54cdd529e53cd4e4b52cbbd019de41bd909e20cca27677ea1732f
Oct 30 20:36:20.760962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:20.760909    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv" event={"ID":"baf1259a-192e-4271-9bdd-d602cbc7a47e","Type":"ContainerStarted","Data":"f59f5b05f87c0c408851927a20458c620f8ca78d9ca10de47bd288edeb13d184"}
Oct 30 20:36:20.761357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:20.761010    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv" event={"ID":"baf1259a-192e-4271-9bdd-d602cbc7a47e","Type":"ContainerStarted","Data":"f9b27c3f9db54cdd529e53cd4e4b52cbbd019de41bd909e20cca27677ea1732f"}
Oct 30 20:36:20.878266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:20.878217    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv" podStartSLOduration=1.878203061 podStartE2EDuration="1.878203061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:20.876124332 +0000 UTC m=+961.476926291" watchObservedRunningTime="2026-10-30 20:36:20.878203061 +0000 UTC m=+961.479005021"
Oct 30 20:36:21.769603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.769566    5967 generic.go:356] "Generic (PLEG): container finished" podID="baf1259a-192e-4271-9bdd-d602cbc7a47e" containerID="f59f5b05f87c0c408851927a20458c620f8ca78d9ca10de47bd288edeb13d184" exitCode=0
Oct 30 20:36:21.770078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.769649    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv" event={"ID":"baf1259a-192e-4271-9bdd-d602cbc7a47e","Type":"ContainerDied","Data":"f59f5b05f87c0c408851927a20458c620f8ca78d9ca10de47bd288edeb13d184"}
Oct 30 20:36:21.776886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.776852    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"]
Oct 30 20:36:21.781113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.781093    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.837998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.837952    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"]
Oct 30 20:36:21.868796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.868718    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.868985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.868842    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96p7\" (UniqueName: \"kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.868985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.868880    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.970390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.970355    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.970390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.970395    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96p7\" (UniqueName: \"kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.970637 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.970420    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.971023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.971003    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:21.972925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:21.972902    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:22.027660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.027579    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96p7\" (UniqueName: \"kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7\") pod \"maas-api-revoke-sub-5dd31e55-bffsm\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:22.094945 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.094912    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:22.243833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.243796    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"]
Oct 30 20:36:22.249283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.249259    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.275431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.275395    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.275598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.275455    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.275690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.275667    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcdd\" (UniqueName: \"kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.305166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.305074    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"]
Oct 30 20:36:22.376528 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.376494    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.376723 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.376616    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcdd\" (UniqueName: \"kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.376723 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.376688    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.377392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.377369    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.379090 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.379068    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.432917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.432882    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcdd\" (UniqueName: \"kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd\") pod \"maas-api-revoke-sub-cb8ed325-p6r88\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.475517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.475489    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"]
Oct 30 20:36:22.476727 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:22.476692    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb3403f_f568_4a98_855d_90be9bc5ecc0.slice/crio-39c8602cd790a0b5af34deb17d2b1da365fb58ca90c3eae659ad64879e89470a WatchSource:0}: Error finding container 39c8602cd790a0b5af34deb17d2b1da365fb58ca90c3eae659ad64879e89470a: Status 404 returned error can't find the container with id 39c8602cd790a0b5af34deb17d2b1da365fb58ca90c3eae659ad64879e89470a
Oct 30 20:36:22.564754 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.564651    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:22.778411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.778375    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm" event={"ID":"0eb3403f-f568-4a98-855d-90be9bc5ecc0","Type":"ContainerStarted","Data":"39c8602cd790a0b5af34deb17d2b1da365fb58ca90c3eae659ad64879e89470a"}
Oct 30 20:36:22.921374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.921346    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:22.946202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.946177    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"]
Oct 30 20:36:22.947217 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:22.947187    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cab2e9_4ec3_4287_8737_c72a6b75db92.slice/crio-8e52970359afc34ed911373f3c4831a8b32752216c84f9f7fcd83550f7fa460a WatchSource:0}: Error finding container 8e52970359afc34ed911373f3c4831a8b32752216c84f9f7fcd83550f7fa460a: Status 404 returned error can't find the container with id 8e52970359afc34ed911373f3c4831a8b32752216c84f9f7fcd83550f7fa460a
Oct 30 20:36:22.981792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.981763    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca\") pod \"baf1259a-192e-4271-9bdd-d602cbc7a47e\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") "
Oct 30 20:36:22.981917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.981905    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p\" (UniqueName: \"kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p\") pod \"baf1259a-192e-4271-9bdd-d602cbc7a47e\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") "
Oct 30 20:36:22.982025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.982009    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca\") pod \"baf1259a-192e-4271-9bdd-d602cbc7a47e\" (UID: \"baf1259a-192e-4271-9bdd-d602cbc7a47e\") "
Oct 30 20:36:22.982361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.982331    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca" pod "baf1259a-192e-4271-9bdd-d602cbc7a47e" (UID: "baf1259a-192e-4271-9bdd-d602cbc7a47e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:22.982506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.982487    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:22.983945 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.983923    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca" pod "baf1259a-192e-4271-9bdd-d602cbc7a47e" (UID: "baf1259a-192e-4271-9bdd-d602cbc7a47e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:22.984259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:22.984237    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p" pod "baf1259a-192e-4271-9bdd-d602cbc7a47e" (UID: "baf1259a-192e-4271-9bdd-d602cbc7a47e"). InnerVolumeSpecName "kube-api-access-zh47p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:23.084758 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.084712    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh47p\" (UniqueName: \"kubernetes.io/projected/baf1259a-192e-4271-9bdd-d602cbc7a47e-kube-api-access-zh47p\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:23.084758 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.084754    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf1259a-192e-4271-9bdd-d602cbc7a47e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:23.786879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.786838    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88" event={"ID":"f1cab2e9-4ec3-4287-8737-c72a6b75db92","Type":"ContainerStarted","Data":"df4b875c9a93f8485b8c9cc89c3259451681069d2ccb8e23b45e767d3d54d513"}
Oct 30 20:36:23.787273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.786897    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88" event={"ID":"f1cab2e9-4ec3-4287-8737-c72a6b75db92","Type":"ContainerStarted","Data":"8e52970359afc34ed911373f3c4831a8b32752216c84f9f7fcd83550f7fa460a"}
Oct 30 20:36:23.788185 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.788161    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"
Oct 30 20:36:23.788343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.788160    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv" event={"ID":"baf1259a-192e-4271-9bdd-d602cbc7a47e","Type":"ContainerDied","Data":"f9b27c3f9db54cdd529e53cd4e4b52cbbd019de41bd909e20cca27677ea1732f"}
Oct 30 20:36:23.788343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.788282    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b27c3f9db54cdd529e53cd4e4b52cbbd019de41bd909e20cca27677ea1732f"
Oct 30 20:36:23.789963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.789929    5967 generic.go:356] "Generic (PLEG): container finished" podID="0eb3403f-f568-4a98-855d-90be9bc5ecc0" containerID="85d6597afe0039229ba24943d1870a7130d859945aee7e06924a21ef82879908" exitCode=0
Oct 30 20:36:23.790086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.790057    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm" event={"ID":"0eb3403f-f568-4a98-855d-90be9bc5ecc0","Type":"ContainerDied","Data":"85d6597afe0039229ba24943d1870a7130d859945aee7e06924a21ef82879908"}
Oct 30 20:36:23.903584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:23.903526    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88" podStartSLOduration=1.9035105030000001 podStartE2EDuration="1.903510503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:23.900206131 +0000 UTC m=+964.501008091" watchObservedRunningTime="2026-10-30 20:36:23.903510503 +0000 UTC m=+964.504312463"
Oct 30 20:36:24.799689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:24.799650    5967 generic.go:356] "Generic (PLEG): container finished" podID="f1cab2e9-4ec3-4287-8737-c72a6b75db92" containerID="df4b875c9a93f8485b8c9cc89c3259451681069d2ccb8e23b45e767d3d54d513" exitCode=0
Oct 30 20:36:24.800206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:24.799765    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88" event={"ID":"f1cab2e9-4ec3-4287-8737-c72a6b75db92","Type":"ContainerDied","Data":"df4b875c9a93f8485b8c9cc89c3259451681069d2ccb8e23b45e767d3d54d513"}
Oct 30 20:36:24.962026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:24.961996    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:25.008867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.008831    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca\") pod \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") "
Oct 30 20:36:25.009066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.008974    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca\") pod \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") "
Oct 30 20:36:25.009147 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.009120    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7\" (UniqueName: \"kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7\") pod \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\" (UID: \"0eb3403f-f568-4a98-855d-90be9bc5ecc0\") "
Oct 30 20:36:25.009324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.009300    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca" pod "0eb3403f-f568-4a98-855d-90be9bc5ecc0" (UID: "0eb3403f-f568-4a98-855d-90be9bc5ecc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:25.009551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.009532    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:25.011131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.011105    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca" pod "0eb3403f-f568-4a98-855d-90be9bc5ecc0" (UID: "0eb3403f-f568-4a98-855d-90be9bc5ecc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:25.011256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.011237    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7" pod "0eb3403f-f568-4a98-855d-90be9bc5ecc0" (UID: "0eb3403f-f568-4a98-855d-90be9bc5ecc0"). InnerVolumeSpecName "kube-api-access-q96p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:25.111425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.111335    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q96p7\" (UniqueName: \"kubernetes.io/projected/0eb3403f-f568-4a98-855d-90be9bc5ecc0-kube-api-access-q96p7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:25.111425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.111379    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb3403f-f568-4a98-855d-90be9bc5ecc0-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:25.780818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.780775    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"]
Oct 30 20:36:25.781346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.781332    5967 cpu_manager.go:414] "containerMap: removing container" podUID="baf1259a-192e-4271-9bdd-d602cbc7a47e" containerName="revoke-keys"
Oct 30 20:36:25.781389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.781347    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf1259a-192e-4271-9bdd-d602cbc7a47e" containerName="revoke-keys"
Oct 30 20:36:25.781389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.781368    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0eb3403f-f568-4a98-855d-90be9bc5ecc0" containerName="revoke-keys"
Oct 30 20:36:25.781389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.781374    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb3403f-f568-4a98-855d-90be9bc5ecc0" containerName="revoke-keys"
Oct 30 20:36:25.781482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.781473    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf1259a-192e-4271-9bdd-d602cbc7a47e" containerName="revoke-keys"
Oct 30 20:36:25.781514 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.781485    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb3403f-f568-4a98-855d-90be9bc5ecc0" containerName="revoke-keys"
Oct 30 20:36:25.785240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.785220    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.808505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.808426    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"
Oct 30 20:36:25.808505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.808447    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm" event={"ID":"0eb3403f-f568-4a98-855d-90be9bc5ecc0","Type":"ContainerDied","Data":"39c8602cd790a0b5af34deb17d2b1da365fb58ca90c3eae659ad64879e89470a"}
Oct 30 20:36:25.808505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.808493    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c8602cd790a0b5af34deb17d2b1da365fb58ca90c3eae659ad64879e89470a"
Oct 30 20:36:25.887375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.887332    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"]
Oct 30 20:36:25.921647 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.921615    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvbs\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.921858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.921787    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.921858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.921816    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.921858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.921839    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.922000 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.921916    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.922000 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.921976    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.922073 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.922033    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.922169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.922152    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.922226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.922184    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:25.974155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:25.974126    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:26.022924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.022894    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd\" (UniqueName: \"kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd\") pod \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") "
Oct 30 20:36:26.023108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.022944    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca\") pod \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") "
Oct 30 20:36:26.023108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023025    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca\") pod \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\" (UID: \"f1cab2e9-4ec3-4287-8737-c72a6b75db92\") "
Oct 30 20:36:26.023201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023166    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023213    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023235    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023259    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvbs\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023336    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca" pod "f1cab2e9-4ec3-4287-8737-c72a6b75db92" (UID: "f1cab2e9-4ec3-4287-8737-c72a6b75db92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:26.023459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023422    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023465    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023491    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023543    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023584    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023586    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.023845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.023823    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.024839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.024026    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.024839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.024059    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:26.024839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.024139    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.024839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.024779    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.025551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.025517    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd" pod "f1cab2e9-4ec3-4287-8737-c72a6b75db92" (UID: "f1cab2e9-4ec3-4287-8737-c72a6b75db92"). InnerVolumeSpecName "kube-api-access-jdcdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:26.025551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.025534    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca" pod "f1cab2e9-4ec3-4287-8737-c72a6b75db92" (UID: "f1cab2e9-4ec3-4287-8737-c72a6b75db92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:26.026028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.026008    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.026324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.026307    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.079566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.079501    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.079794 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.079773    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvbs\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jjg48\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.103399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.098579    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:26.124774 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.124718    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcdd\" (UniqueName: \"kubernetes.io/projected/f1cab2e9-4ec3-4287-8737-c72a6b75db92-kube-api-access-jdcdd\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:26.124937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.124812    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1cab2e9-4ec3-4287-8737-c72a6b75db92-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:26.531150 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.531115    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"]
Oct 30 20:36:26.531698 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:26.531671    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f553a9_eae7_44da_abf0_5f6181ea9656.slice/crio-7a8d5d49aaffe45780c3297077778f365cbe332489f3f00e92b92373a1a7babe WatchSource:0}: Error finding container 7a8d5d49aaffe45780c3297077778f365cbe332489f3f00e92b92373a1a7babe: Status 404 returned error can't find the container with id 7a8d5d49aaffe45780c3297077778f365cbe332489f3f00e92b92373a1a7babe
Oct 30 20:36:26.534024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.533989    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:26.534105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.534049    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:26.534105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.534077    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:36:26.819348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.819235    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" event={"ID":"74f553a9-eae7-44da-abf0-5f6181ea9656","Type":"ContainerStarted","Data":"739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7"}
Oct 30 20:36:26.819348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.819306    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" event={"ID":"74f553a9-eae7-44da-abf0-5f6181ea9656","Type":"ContainerStarted","Data":"7a8d5d49aaffe45780c3297077778f365cbe332489f3f00e92b92373a1a7babe"}
Oct 30 20:36:26.820952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.820929    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88" event={"ID":"f1cab2e9-4ec3-4287-8737-c72a6b75db92","Type":"ContainerDied","Data":"8e52970359afc34ed911373f3c4831a8b32752216c84f9f7fcd83550f7fa460a"}
Oct 30 20:36:26.821067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.820974    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"
Oct 30 20:36:26.821067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.821034    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e52970359afc34ed911373f3c4831a8b32752216c84f9f7fcd83550f7fa460a"
Oct 30 20:36:26.938222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:26.938170    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podStartSLOduration=1.938154505 podStartE2EDuration="1.938154505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:26.936318397 +0000 UTC m=+967.537120356" watchObservedRunningTime="2026-10-30 20:36:26.938154505 +0000 UTC m=+967.538956464"
Oct 30 20:36:27.099732 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:27.099642    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:27.101681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:27.101656    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:27.101834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:27.101706    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:28.100005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:28.099956    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:28.100490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:28.100013    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:29.099670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:29.099634    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:29.099888 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:29.099696    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:30.099460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:30.099414    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:30.099882 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:30.099479    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:31.099688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:31.099654    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:31.100118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:31.099709    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:32.004958 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:36:32.004920    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:36:32.100088 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:32.100047    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:32.100589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:32.100108    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:33.029083 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.029050    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"]
Oct 30 20:36:33.029759 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.029726    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f1cab2e9-4ec3-4287-8737-c72a6b75db92" containerName="revoke-keys"
Oct 30 20:36:33.029759 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.029761    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cab2e9-4ec3-4287-8737-c72a6b75db92" containerName="revoke-keys"
Oct 30 20:36:33.029991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.029978    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cab2e9-4ec3-4287-8737-c72a6b75db92" containerName="revoke-keys"
Oct 30 20:36:33.035153 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.035129    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.091285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.091245    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"]
Oct 30 20:36:33.099771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.099715    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:36:33.099963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.099814    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:36:33.135422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.135384    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.135849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.135596    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.135849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.135714    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt5f\" (UniqueName: \"kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.135849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.135771    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:33.136020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.136004    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:36:33.136098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.136078    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:33.136889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.136873    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:33.236972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.236939    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.237187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.237053    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.237187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.237110    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt5f\" (UniqueName: \"kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.237869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.237840    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.239530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.239508    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.294196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.294123    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt5f\" (UniqueName: \"kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f\") pod \"maas-api-revoke-sub-30c80372-s9k78\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.351500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.351467    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:33.734893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.734867    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"]
Oct 30 20:36:33.735945 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:33.735920    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7156d30b_5884_4ea5_9b0d_4e37a6b6db02.slice/crio-bb9c74cea5eb9a0ebdad392ffc7a66eb798cd859e6166f63885b1b0dd926823d WatchSource:0}: Error finding container bb9c74cea5eb9a0ebdad392ffc7a66eb798cd859e6166f63885b1b0dd926823d: Status 404 returned error can't find the container with id bb9c74cea5eb9a0ebdad392ffc7a66eb798cd859e6166f63885b1b0dd926823d
Oct 30 20:36:33.877704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:33.877674    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78" event={"ID":"7156d30b-5884-4ea5-9b0d-4e37a6b6db02","Type":"ContainerStarted","Data":"bb9c74cea5eb9a0ebdad392ffc7a66eb798cd859e6166f63885b1b0dd926823d"}
Oct 30 20:36:34.715967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:34.715930    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:34.716690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:34.716645    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:34.717716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:34.717696    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:36:34.886430 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:34.886392    5967 generic.go:356] "Generic (PLEG): container finished" podID="7156d30b-5884-4ea5-9b0d-4e37a6b6db02" containerID="c69086a28b814f22971d950830b319e256ae27f030bc94623fe265fe18d7a0f2" exitCode=0
Oct 30 20:36:34.886621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:34.886467    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78" event={"ID":"7156d30b-5884-4ea5-9b0d-4e37a6b6db02","Type":"ContainerDied","Data":"c69086a28b814f22971d950830b319e256ae27f030bc94623fe265fe18d7a0f2"}
Oct 30 20:36:36.048062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.048037    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:36.070023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.069992    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca\") pod \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") "
Oct 30 20:36:36.070194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.070069    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f\" (UniqueName: \"kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f\") pod \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") "
Oct 30 20:36:36.070194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.070096    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca\") pod \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\" (UID: \"7156d30b-5884-4ea5-9b0d-4e37a6b6db02\") "
Oct 30 20:36:36.070504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.070482    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca" pod "7156d30b-5884-4ea5-9b0d-4e37a6b6db02" (UID: "7156d30b-5884-4ea5-9b0d-4e37a6b6db02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:36.072107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.072082    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca" pod "7156d30b-5884-4ea5-9b0d-4e37a6b6db02" (UID: "7156d30b-5884-4ea5-9b0d-4e37a6b6db02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:36.072334 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.072311    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f" pod "7156d30b-5884-4ea5-9b0d-4e37a6b6db02" (UID: "7156d30b-5884-4ea5-9b0d-4e37a6b6db02"). InnerVolumeSpecName "kube-api-access-2pt5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:36.171950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.171864    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:36.171950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.171900    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt5f\" (UniqueName: \"kubernetes.io/projected/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-kube-api-access-2pt5f\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:36.171950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.171915    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7156d30b-5884-4ea5-9b0d-4e37a6b6db02-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:36.903418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.903388    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"
Oct 30 20:36:36.903587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.903412    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78" event={"ID":"7156d30b-5884-4ea5-9b0d-4e37a6b6db02","Type":"ContainerDied","Data":"bb9c74cea5eb9a0ebdad392ffc7a66eb798cd859e6166f63885b1b0dd926823d"}
Oct 30 20:36:36.903587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:36.903470    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9c74cea5eb9a0ebdad392ffc7a66eb798cd859e6166f63885b1b0dd926823d"
Oct 30 20:36:37.684473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.684439    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"]
Oct 30 20:36:37.685030 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.685016    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7156d30b-5884-4ea5-9b0d-4e37a6b6db02" containerName="revoke-keys"
Oct 30 20:36:37.685071 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.685031    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7156d30b-5884-4ea5-9b0d-4e37a6b6db02" containerName="revoke-keys"
Oct 30 20:36:37.685156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.685146    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7156d30b-5884-4ea5-9b0d-4e37a6b6db02" containerName="revoke-keys"
Oct 30 20:36:37.688712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.688693    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.743041 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.743009    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:37.743286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.743054    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:37.788075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.788037    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852vl\" (UniqueName: \"kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.788253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.788160    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.788253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.788237    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.792444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.792409    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:36:37.792585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.792570    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:37.796561 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.796535    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"]
Oct 30 20:36:37.889371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.889321    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852vl\" (UniqueName: \"kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.889578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.889427    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.889578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.889514    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.890253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.890226    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.892049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.892025    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:37.946923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:37.946830    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-852vl\" (UniqueName: \"kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl\") pod \"maas-api-revoke-sub-918014c3-txqpx\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:38.002220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:38.002189    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:38.382469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:38.382441    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"]
Oct 30 20:36:38.383324 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:38.383298    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7517642_8816_485b_96e1_fad2f8198748.slice/crio-5c3aed3f903c35d1d90add21942f573ebb5c684a7188e693fe034b01550470b3 WatchSource:0}: Error finding container 5c3aed3f903c35d1d90add21942f573ebb5c684a7188e693fe034b01550470b3: Status 404 returned error can't find the container with id 5c3aed3f903c35d1d90add21942f573ebb5c684a7188e693fe034b01550470b3
Oct 30 20:36:38.920020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:38.919985    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx" event={"ID":"e7517642-8816-485b-96e1-fad2f8198748","Type":"ContainerStarted","Data":"1ab5daef454ee0796d0c47a7960d0b3535d54a0a672336c9f13859fad945d209"}
Oct 30 20:36:38.920511 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:38.920067    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx" event={"ID":"e7517642-8816-485b-96e1-fad2f8198748","Type":"ContainerStarted","Data":"5c3aed3f903c35d1d90add21942f573ebb5c684a7188e693fe034b01550470b3"}
Oct 30 20:36:39.037714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:39.037661    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx" podStartSLOduration=2.037647529 podStartE2EDuration="2.037647529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:39.034043382 +0000 UTC m=+979.634845341" watchObservedRunningTime="2026-10-30 20:36:39.037647529 +0000 UTC m=+979.638449488"
Oct 30 20:36:39.927948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:39.927909    5967 generic.go:356] "Generic (PLEG): container finished" podID="e7517642-8816-485b-96e1-fad2f8198748" containerID="1ab5daef454ee0796d0c47a7960d0b3535d54a0a672336c9f13859fad945d209" exitCode=0
Oct 30 20:36:39.928337 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:39.927998    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx" event={"ID":"e7517642-8816-485b-96e1-fad2f8198748","Type":"ContainerDied","Data":"1ab5daef454ee0796d0c47a7960d0b3535d54a0a672336c9f13859fad945d209"}
Oct 30 20:36:41.187562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.187514    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:41.252185 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.252146    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca\") pod \"e7517642-8816-485b-96e1-fad2f8198748\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") "
Oct 30 20:36:41.252372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.252343    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl\" (UniqueName: \"kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl\") pod \"e7517642-8816-485b-96e1-fad2f8198748\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") "
Oct 30 20:36:41.252477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.252460    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca\") pod \"e7517642-8816-485b-96e1-fad2f8198748\" (UID: \"e7517642-8816-485b-96e1-fad2f8198748\") "
Oct 30 20:36:41.252538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.252477    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca" pod "e7517642-8816-485b-96e1-fad2f8198748" (UID: "e7517642-8816-485b-96e1-fad2f8198748"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:41.252946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.252926    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7517642-8816-485b-96e1-fad2f8198748-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:41.254641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.254616    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl" pod "e7517642-8816-485b-96e1-fad2f8198748" (UID: "e7517642-8816-485b-96e1-fad2f8198748"). InnerVolumeSpecName "kube-api-access-852vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:41.254720 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.254691    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca" pod "e7517642-8816-485b-96e1-fad2f8198748" (UID: "e7517642-8816-485b-96e1-fad2f8198748"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:41.354626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.354594    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-852vl\" (UniqueName: \"kubernetes.io/projected/e7517642-8816-485b-96e1-fad2f8198748-kube-api-access-852vl\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:41.354626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.354623    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7517642-8816-485b-96e1-fad2f8198748-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:41.946379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.946290    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"
Oct 30 20:36:41.946379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.946323    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx" event={"ID":"e7517642-8816-485b-96e1-fad2f8198748","Type":"ContainerDied","Data":"5c3aed3f903c35d1d90add21942f573ebb5c684a7188e693fe034b01550470b3"}
Oct 30 20:36:41.946379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:41.946369    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3aed3f903c35d1d90add21942f573ebb5c684a7188e693fe034b01550470b3"
Oct 30 20:36:48.969991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:48.969958    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"]
Oct 30 20:36:48.970716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:48.970697    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e7517642-8816-485b-96e1-fad2f8198748" containerName="revoke-keys"
Oct 30 20:36:48.970716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:48.970716    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7517642-8816-485b-96e1-fad2f8198748" containerName="revoke-keys"
Oct 30 20:36:48.970873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:48.970856    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7517642-8816-485b-96e1-fad2f8198748" containerName="revoke-keys"
Oct 30 20:36:48.974527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:48.974510    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.031479 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.031443    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"]
Oct 30 20:36:49.039608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.039578    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlklk\" (UniqueName: \"kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.039778 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.039658    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.039866 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.039779    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.076364 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.076336    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:49.076516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.076374    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:49.076516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.076417    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:49.076516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.076508    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:36:49.141042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.141012    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlklk\" (UniqueName: \"kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.141174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.141064    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.141174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.141092    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.141660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.141642    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.143472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.143453    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.197279 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.197257    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlklk\" (UniqueName: \"kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk\") pod \"maas-api-revoke-sub-b9539494-5tfrq\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.292402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.292344    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:49.666962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:49.666932    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"]
Oct 30 20:36:49.667231 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:36:49.667206    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076824e6_bf71_47d3_bdfa_6faf285dd3b9.slice/crio-7705933c70b2746ecd5906b148e2af738f8ad2a3383ed9079e84956b480fbc0f WatchSource:0}: Error finding container 7705933c70b2746ecd5906b148e2af738f8ad2a3383ed9079e84956b480fbc0f: Status 404 returned error can't find the container with id 7705933c70b2746ecd5906b148e2af738f8ad2a3383ed9079e84956b480fbc0f
Oct 30 20:36:50.002560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:50.002531    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq" event={"ID":"076824e6-bf71-47d3-bdfa-6faf285dd3b9","Type":"ContainerStarted","Data":"235c8ff4a3399ac09f86176ccce684765d7ab2ce155afe8257a2947f1e58a6b0"}
Oct 30 20:36:50.002885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:50.002586    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq" event={"ID":"076824e6-bf71-47d3-bdfa-6faf285dd3b9","Type":"ContainerStarted","Data":"7705933c70b2746ecd5906b148e2af738f8ad2a3383ed9079e84956b480fbc0f"}
Oct 30 20:36:50.124621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:50.124568    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq" podStartSLOduration=2.124554934 podStartE2EDuration="2.124554934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:50.121054402 +0000 UTC m=+990.721856363" watchObservedRunningTime="2026-10-30 20:36:50.124554934 +0000 UTC m=+990.725356894"
Oct 30 20:36:51.010502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:51.010464    5967 generic.go:356] "Generic (PLEG): container finished" podID="076824e6-bf71-47d3-bdfa-6faf285dd3b9" containerID="235c8ff4a3399ac09f86176ccce684765d7ab2ce155afe8257a2947f1e58a6b0" exitCode=0
Oct 30 20:36:51.010936 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:51.010559    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq" event={"ID":"076824e6-bf71-47d3-bdfa-6faf285dd3b9","Type":"ContainerDied","Data":"235c8ff4a3399ac09f86176ccce684765d7ab2ce155afe8257a2947f1e58a6b0"}
Oct 30 20:36:52.169458 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.169433    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:52.276488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.276427    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca\") pod \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") "
Oct 30 20:36:52.276606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.276494    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca\") pod \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") "
Oct 30 20:36:52.276606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.276568    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk\" (UniqueName: \"kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk\") pod \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\" (UID: \"076824e6-bf71-47d3-bdfa-6faf285dd3b9\") "
Oct 30 20:36:52.276816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.276777    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca" pod "076824e6-bf71-47d3-bdfa-6faf285dd3b9" (UID: "076824e6-bf71-47d3-bdfa-6faf285dd3b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:52.277091 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.277070    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:52.278623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.278596    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk" pod "076824e6-bf71-47d3-bdfa-6faf285dd3b9" (UID: "076824e6-bf71-47d3-bdfa-6faf285dd3b9"). InnerVolumeSpecName "kube-api-access-xlklk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:52.278704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.278666    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca" pod "076824e6-bf71-47d3-bdfa-6faf285dd3b9" (UID: "076824e6-bf71-47d3-bdfa-6faf285dd3b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:52.378420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.378393    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076824e6-bf71-47d3-bdfa-6faf285dd3b9-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:52.378420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:52.378417    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlklk\" (UniqueName: \"kubernetes.io/projected/076824e6-bf71-47d3-bdfa-6faf285dd3b9-kube-api-access-xlklk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:53.032446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:53.032417    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"
Oct 30 20:36:53.032446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:53.032432    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq" event={"ID":"076824e6-bf71-47d3-bdfa-6faf285dd3b9","Type":"ContainerDied","Data":"7705933c70b2746ecd5906b148e2af738f8ad2a3383ed9079e84956b480fbc0f"}
Oct 30 20:36:53.032733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:36:53.032492    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7705933c70b2746ecd5906b148e2af738f8ad2a3383ed9079e84956b480fbc0f"
Oct 30 20:36:53.085942 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:36:53.085916    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076824e6_bf71_47d3_bdfa_6faf285dd3b9.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:37:01.980404 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:37:01.980323    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:37:27.882483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.882450    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"]
Oct 30 20:37:27.885168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.883030    5967 cpu_manager.go:414] "containerMap: removing container" podUID="076824e6-bf71-47d3-bdfa-6faf285dd3b9" containerName="revoke-keys"
Oct 30 20:37:27.885168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.883042    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076824e6-bf71-47d3-bdfa-6faf285dd3b9" containerName="revoke-keys"
Oct 30 20:37:27.885168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.883171    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="076824e6-bf71-47d3-bdfa-6faf285dd3b9" containerName="revoke-keys"
Oct 30 20:37:27.886469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.886453    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.893503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.893202    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.893503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.893258    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.893503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.893377    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zpm\" (UniqueName: \"kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.893694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.893546    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.893694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.893608    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.893694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.893676    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.945482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.945448    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"]
Oct 30 20:37:27.989712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.989677    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"distinct-w4-8e2fc1-kserve-self-signed-certs\""
Oct 30 20:37:27.995182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995150    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995194    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995224    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zpm\" (UniqueName: \"kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995310    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995392    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995435    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995563    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995639    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.995731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.995716    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.997626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.997600    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:27.997813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:27.997797    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:28.061649 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:28.061594    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zpm\" (UniqueName: \"kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm\") pod \"distinct-w4-8e2fc1-kserve-886c4b4db-psnh2\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:28.198439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:28.198355    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:28.577140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:28.577108    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"]
Oct 30 20:37:28.577510 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:37:28.577485    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b9a740f_ff20_45ba_a128_24f890436aa4.slice/crio-0b7bcadeba41951758a5d5c04ed6ff895a35abeeb913b38ba3e5745e6acfb73a WatchSource:0}: Error finding container 0b7bcadeba41951758a5d5c04ed6ff895a35abeeb913b38ba3e5745e6acfb73a: Status 404 returned error can't find the container with id 0b7bcadeba41951758a5d5c04ed6ff895a35abeeb913b38ba3e5745e6acfb73a
Oct 30 20:37:29.317990 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:29.317954    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" event={"ID":"3b9a740f-ff20-45ba-a128-24f890436aa4","Type":"ContainerStarted","Data":"06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509"}
Oct 30 20:37:29.318397 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:29.317996    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" event={"ID":"3b9a740f-ff20-45ba-a128-24f890436aa4","Type":"ContainerStarted","Data":"0b7bcadeba41951758a5d5c04ed6ff895a35abeeb913b38ba3e5745e6acfb73a"}
Oct 30 20:37:29.433558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:29.433504    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podStartSLOduration=2.433488648 podStartE2EDuration="2.433488648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:29.432542865 +0000 UTC m=+1030.033344848" watchObservedRunningTime="2026-10-30 20:37:29.433488648 +0000 UTC m=+1030.034290608"
Oct 30 20:37:30.198948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:30.198918    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:31.981536 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:37:31.981504    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:37:34.855619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.855585    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"]
Oct 30 20:37:34.859665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.859650    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:34.915941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.915913    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"openshift-service-ca.crt\""
Oct 30 20:37:34.917089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.917065    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"]
Oct 30 20:37:34.962062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.962042    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"default-dockercfg-6bjt2\""
Oct 30 20:37:34.962180 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.962141    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"distinct-w1-5f9b95-kserve-self-signed-certs\""
Oct 30 20:37:34.968673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.968654    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:34.968779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.968689    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464j8\" (UniqueName: \"kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:34.968779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.968723    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:34.968880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.968848    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:34.968928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.968884    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:34.968928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:34.968901    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.013452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.013425    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"kube-root-ca.crt\""
Oct 30 20:37:35.070915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.070885    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.070943    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464j8\" (UniqueName: \"kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.070994    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.071025    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.071070    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.071231    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.071297    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071424 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.071338    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.071424 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.071390    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.073462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.073440    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.073607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.073586    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.127622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.127577    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464j8\" (UniqueName: \"kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8\") pod \"distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") " pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.172483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.172456    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:35.559882 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:35.559850    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"]
Oct 30 20:37:35.560153 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:37:35.560133    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3b80af_9651_40cc_95aa_293d7d0ac560.slice/crio-1d8d2aa21148a8a962acaa3b8e0e2398ef5d17e09456bc75b3ea0f31b4d4f9a9 WatchSource:0}: Error finding container 1d8d2aa21148a8a962acaa3b8e0e2398ef5d17e09456bc75b3ea0f31b4d4f9a9: Status 404 returned error can't find the container with id 1d8d2aa21148a8a962acaa3b8e0e2398ef5d17e09456bc75b3ea0f31b4d4f9a9
Oct 30 20:37:36.365634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:36.365596    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" event={"ID":"3c3b80af-9651-40cc-95aa-293d7d0ac560","Type":"ContainerStarted","Data":"7a7b4c6a7ff873103f80594034bf105f88560fbf6deb1c0b8dc4f6e60c91dbed"}
Oct 30 20:37:36.365634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:36.365636    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" event={"ID":"3c3b80af-9651-40cc-95aa-293d7d0ac560","Type":"ContainerStarted","Data":"1d8d2aa21148a8a962acaa3b8e0e2398ef5d17e09456bc75b3ea0f31b4d4f9a9"}
Oct 30 20:37:36.497780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:36.497719    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podStartSLOduration=2.497706104 podStartE2EDuration="2.497706104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:36.496025898 +0000 UTC m=+1037.096827860" watchObservedRunningTime="2026-10-30 20:37:36.497706104 +0000 UTC m=+1037.098508064"
Oct 30 20:37:37.173370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:37.173324    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:38.198723 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:38.198690    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:38.211420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:38.211400    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:38.389176 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:38.389142    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:37:45.173135 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:45.173107    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:45.185448 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:45.185419    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:45.437534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:45.437452    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:37:53.396697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.396647    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"]
Oct 30 20:37:53.404580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.404553    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.509349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.509309    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"]
Oct 30 20:37:53.586570 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586541    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv22v\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.586753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586604    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.586753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586687    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.586880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586770    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.586880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586831    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.586944 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586923    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.586982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.586953    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.587026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.587013    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.587071 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.587042    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688394    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688435    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688487    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688508    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688547    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688574    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.688690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688644    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv22v\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688707    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688771    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.688865    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.689119    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.689157    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.689233    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.689603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.689583    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.691047 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.691028    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.691352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.691332    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.745429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.745398    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:53.745575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:53.745474    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv22v\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:54.018148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:54.018121    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:54.450414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:54.450384    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"]
Oct 30 20:37:54.450720 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:37:54.450526    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68229717_55df_40ff_9c71_1aa4daa43ae8.slice/crio-0a1147b973d95d06ec6548df55f836f3f6258c1b77ca2047b59fff403b4eda35 WatchSource:0}: Error finding container 0a1147b973d95d06ec6548df55f836f3f6258c1b77ca2047b59fff403b4eda35: Status 404 returned error can't find the container with id 0a1147b973d95d06ec6548df55f836f3f6258c1b77ca2047b59fff403b4eda35
Oct 30 20:37:54.452876 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:54.452840    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:54.452968 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:54.452926    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:54.453040 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:54.452970    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:54.495187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:54.495159    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" event={"ID":"68229717-55df-40ff-9c71-1aa4daa43ae8","Type":"ContainerStarted","Data":"0a1147b973d95d06ec6548df55f836f3f6258c1b77ca2047b59fff403b4eda35"}
Oct 30 20:37:55.504281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:55.504245    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" event={"ID":"68229717-55df-40ff-9c71-1aa4daa43ae8","Type":"ContainerStarted","Data":"62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd"}
Oct 30 20:37:55.736007 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:55.735955    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podStartSLOduration=2.735940859 podStartE2EDuration="2.735940859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:55.683187782 +0000 UTC m=+1056.283989742" watchObservedRunningTime="2026-10-30 20:37:55.735940859 +0000 UTC m=+1056.336742819"
Oct 30 20:37:56.018730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:56.018698    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:37:56.020603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:56.020577    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:37:56.020755 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:56.020621    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:37:57.018815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:57.018777    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:37:57.019325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:57.018828    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:37:58.018478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:58.018446    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:37:58.018681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:58.018501    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:37:59.018556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:59.018524    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:37:59.019076 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:37:59.018587    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:38:00.018453 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:38:00.018414    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:38:00.018649 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:38:00.018476    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:38:01.242512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:38:01.242481    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:38:01.243454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:38:01.243424    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:38:01.246009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:38:01.245987    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 20:38:01.979613 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:38:01.979571    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:38:31.981354 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:38:31.981273    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:39:01.979642 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:39:01.979608    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:39:21.179569 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.179535    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"]
Oct 30 20:39:21.183697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.183682    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.211897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.211877    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.212007 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.211914    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc9x\" (UniqueName: \"kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.212058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.212017    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.285169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.285130    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:39:21.285331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.285179    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:39:21.285331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.285195    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:39:21.285331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.285240    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:39:21.289133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.289108    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"]
Oct 30 20:39:21.313280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.313256    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.313395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.313296    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc9x\" (UniqueName: \"kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.313395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.313373    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.314059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.314042    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.315661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.315644    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.371293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.371264    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc9x\" (UniqueName: \"kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x\") pod \"maas-api-revoke-sub-2a59f45b-tjmg6\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.497380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.497350    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:21.873851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:21.873824    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"]
Oct 30 20:39:21.874971 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:39:21.874940    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd387b4_4fa9_46f1_9e40_0c280cbe2a5e.slice/crio-514c3e35940123195fbdd83b498ec41b01ad951ee19ae6834dc5c93c461eff9e WatchSource:0}: Error finding container 514c3e35940123195fbdd83b498ec41b01ad951ee19ae6834dc5c93c461eff9e: Status 404 returned error can't find the container with id 514c3e35940123195fbdd83b498ec41b01ad951ee19ae6834dc5c93c461eff9e
Oct 30 20:39:22.110097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:22.110068    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6" event={"ID":"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e","Type":"ContainerStarted","Data":"514c3e35940123195fbdd83b498ec41b01ad951ee19ae6834dc5c93c461eff9e"}
Oct 30 20:39:23.118086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:23.118051    5967 generic.go:356] "Generic (PLEG): container finished" podID="2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" containerID="15a92de5edfd6d5d0972b77e022fa385befd8732865db223a102214d4b887645" exitCode=0
Oct 30 20:39:23.118526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:23.118182    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6" event={"ID":"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e","Type":"ContainerDied","Data":"15a92de5edfd6d5d0972b77e022fa385befd8732865db223a102214d4b887645"}
Oct 30 20:39:24.283430 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.283404    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:24.348838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.348807    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x\" (UniqueName: \"kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x\") pod \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") "
Oct 30 20:39:24.348991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.348888    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca\") pod \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") "
Oct 30 20:39:24.348991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.348948    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca\") pod \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\" (UID: \"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e\") "
Oct 30 20:39:24.349410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.349382    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca" pod "2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" (UID: "2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:24.349504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.349471    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:24.351598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.351574    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca" pod "2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" (UID: "2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:24.351598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.351594    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x" pod "2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" (UID: "2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e"). InnerVolumeSpecName "kube-api-access-2mc9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:24.450995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.450933    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mc9x\" (UniqueName: \"kubernetes.io/projected/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-kube-api-access-2mc9x\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:24.450995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:24.450959    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:25.132792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:25.132766    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"
Oct 30 20:39:25.132792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:25.132785    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6" event={"ID":"2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e","Type":"ContainerDied","Data":"514c3e35940123195fbdd83b498ec41b01ad951ee19ae6834dc5c93c461eff9e"}
Oct 30 20:39:25.132997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:25.132837    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="514c3e35940123195fbdd83b498ec41b01ad951ee19ae6834dc5c93c461eff9e"
Oct 30 20:39:31.980790 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:39:31.980726    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:39:33.861262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.861224    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"]
Oct 30 20:39:33.861842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.861827    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" containerName="revoke-keys"
Oct 30 20:39:33.861900 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.861842    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" containerName="revoke-keys"
Oct 30 20:39:33.861957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.861948    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" containerName="revoke-keys"
Oct 30 20:39:33.865478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.865464    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.923873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.923850    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"openshift-service-ca.crt\""
Oct 30 20:39:33.968510 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.968486    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"kube-root-ca.crt\""
Oct 30 20:39:33.968676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.968587    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"default-dockercfg-7xnk5\""
Oct 30 20:39:33.968676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.968662    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"distinct-2-w2-df031b-kserve-self-signed-certs\""
Oct 30 20:39:33.970101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.970082    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.970161 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.970121    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.970282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.970265    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.970420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.970407    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfl5\" (UniqueName: \"kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.970480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.970468    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.970530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.970501    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:33.972615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:33.972597    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"]
Oct 30 20:39:34.071321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071281    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfl5\" (UniqueName: \"kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071378    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071416    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071455    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071493    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071566    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071870    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071944    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.071988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.071975    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.073962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.073938    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.074109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.074093    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.128604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.128529    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfl5\" (UniqueName: \"kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5\") pod \"distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") " pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.177324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.177296    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:34.579647 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:34.579624    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"]
Oct 30 20:39:34.580294 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:39:34.580265    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3d4df14_1364_4b11_8e9e_5adb5289b607.slice/crio-c23cd7542b3f8d3182313b08c3dd9d0863c5f53e7331137b3f0941e035445e0b WatchSource:0}: Error finding container c23cd7542b3f8d3182313b08c3dd9d0863c5f53e7331137b3f0941e035445e0b: Status 404 returned error can't find the container with id c23cd7542b3f8d3182313b08c3dd9d0863c5f53e7331137b3f0941e035445e0b
Oct 30 20:39:35.203334 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:35.203292    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" event={"ID":"d3d4df14-1364-4b11-8e9e-5adb5289b607","Type":"ContainerStarted","Data":"cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e"}
Oct 30 20:39:35.203334 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:35.203334    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" event={"ID":"d3d4df14-1364-4b11-8e9e-5adb5289b607","Type":"ContainerStarted","Data":"c23cd7542b3f8d3182313b08c3dd9d0863c5f53e7331137b3f0941e035445e0b"}
Oct 30 20:39:35.321379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:35.321332    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podStartSLOduration=2.321317713 podStartE2EDuration="2.321317713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:35.319775398 +0000 UTC m=+1155.920577359" watchObservedRunningTime="2026-10-30 20:39:35.321317713 +0000 UTC m=+1155.922119672"
Oct 30 20:39:36.178006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:36.177973    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:37.069930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:37.069903    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"]
Oct 30 20:39:37.119114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:37.119085    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b52ab98-h48p5"]
Oct 30 20:39:37.542664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:37.542621    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda903e2-dd7c-4f1d-b4c7-d878affd0e72" path="/var/lib/kubelet/pods/cda903e2-dd7c-4f1d-b4c7-d878affd0e72/volumes"
Oct 30 20:39:41.864060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.864030    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"]
Oct 30 20:39:41.868437 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.868420    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.938582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.938551    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"]
Oct 30 20:39:41.962677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.962652    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.962832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.962691    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.962873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.962831    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.962873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.962866    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.962987 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.962974    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.963053 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.963041    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfj4\" (UniqueName: \"kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:41.972712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:41.972693    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e2fc1\"/\"distinct-2-w4-8e2fc1-kserve-self-signed-certs\""
Oct 30 20:39:42.064151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064069    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064146    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfj4\" (UniqueName: \"kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064190    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064215    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064266    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064310    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064456    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064610    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.064786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.064669    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.066532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.066513    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.066802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.066786    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.123396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.123372    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfj4\" (UniqueName: \"kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4\") pod \"distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") " pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.181046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.181012    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:42.600871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:42.600846    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"]
Oct 30 20:39:42.601011 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:39:42.600944    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73da3e8a_bc34_4aa6_91e2_61f34e6271dc.slice/crio-d73ac614a13f448828b302b300eaf8ccaa96039bd8c3370152dddab251c5a071 WatchSource:0}: Error finding container d73ac614a13f448828b302b300eaf8ccaa96039bd8c3370152dddab251c5a071: Status 404 returned error can't find the container with id d73ac614a13f448828b302b300eaf8ccaa96039bd8c3370152dddab251c5a071
Oct 30 20:39:43.281994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:43.281960    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" event={"ID":"73da3e8a-bc34-4aa6-91e2-61f34e6271dc","Type":"ContainerStarted","Data":"92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1"}
Oct 30 20:39:43.282458 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:43.282007    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" event={"ID":"73da3e8a-bc34-4aa6-91e2-61f34e6271dc","Type":"ContainerStarted","Data":"d73ac614a13f448828b302b300eaf8ccaa96039bd8c3370152dddab251c5a071"}
Oct 30 20:39:43.404751 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:43.404673    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podStartSLOduration=2.404651996 podStartE2EDuration="2.404651996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:43.399712277 +0000 UTC m=+1164.000514248" watchObservedRunningTime="2026-10-30 20:39:43.404651996 +0000 UTC m=+1164.005453957"
Oct 30 20:39:44.178521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:44.178475    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:44.181468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:44.181428    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:44.206568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:44.206242    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:44.361803 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:44.361760    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:39:52.054222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:52.054184    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"]
Oct 30 20:39:52.102787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:52.102734    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2707831-zdncc"]
Oct 30 20:39:52.181433 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:52.181404    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:52.193568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:52.193547    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:52.388608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:52.388530    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:39:53.442524 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.442498    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"]
Oct 30 20:39:53.447233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.447212    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.508833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.508798    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jcz5\" (UniqueName: \"kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.509001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.508974    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.509056 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.509010    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.544300 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.544265    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8769f3-120f-49bf-9f4c-87a3a9f65b06" path="/var/lib/kubelet/pods/1a8769f3-120f-49bf-9f4c-87a3a9f65b06/volumes"
Oct 30 20:39:53.551403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.551377    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:39:53.551587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.551523    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:39:53.551587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.551548    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:39:53.551587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.551570    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:39:53.555237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.555215    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"]
Oct 30 20:39:53.610205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.610168    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jcz5\" (UniqueName: \"kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.610391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.610265    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.610391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.610293    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.610902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.610881    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.612861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.612833    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.668974 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.668947    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jcz5\" (UniqueName: \"kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5\") pod \"maas-api-revoke-sub-966d45c1-z4l7j\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:53.762859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:53.762838    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:54.151943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:54.151917    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"]
Oct 30 20:39:54.153911 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:39:54.153885    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422bc0c3_948f_4499_a095_0593e0582788.slice/crio-5821a5e72abaef48866bf442cb03eca78e6e9ef5b887d91b9d1d5f9bd18b7df7 WatchSource:0}: Error finding container 5821a5e72abaef48866bf442cb03eca78e6e9ef5b887d91b9d1d5f9bd18b7df7: Status 404 returned error can't find the container with id 5821a5e72abaef48866bf442cb03eca78e6e9ef5b887d91b9d1d5f9bd18b7df7
Oct 30 20:39:54.396058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:54.396020    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j" event={"ID":"422bc0c3-948f-4499-a095-0593e0582788","Type":"ContainerStarted","Data":"5821a5e72abaef48866bf442cb03eca78e6e9ef5b887d91b9d1d5f9bd18b7df7"}
Oct 30 20:39:55.404654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:55.404570    5967 generic.go:356] "Generic (PLEG): container finished" podID="422bc0c3-948f-4499-a095-0593e0582788" containerID="15e2faff4291e9973cf9ef9035ff0a1322fcee123f12db1e3e9242d19c42adc7" exitCode=0
Oct 30 20:39:55.404654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:55.404645    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j" event={"ID":"422bc0c3-948f-4499-a095-0593e0582788","Type":"ContainerDied","Data":"15e2faff4291e9973cf9ef9035ff0a1322fcee123f12db1e3e9242d19c42adc7"}
Oct 30 20:39:56.563551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.563529    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:56.645683 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.645659    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5\" (UniqueName: \"kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5\") pod \"422bc0c3-948f-4499-a095-0593e0582788\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") "
Oct 30 20:39:56.645833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.645790    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca\") pod \"422bc0c3-948f-4499-a095-0593e0582788\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") "
Oct 30 20:39:56.645833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.645814    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca\") pod \"422bc0c3-948f-4499-a095-0593e0582788\" (UID: \"422bc0c3-948f-4499-a095-0593e0582788\") "
Oct 30 20:39:56.647203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.647175    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca" pod "422bc0c3-948f-4499-a095-0593e0582788" (UID: "422bc0c3-948f-4499-a095-0593e0582788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:56.648111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.648089    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca" pod "422bc0c3-948f-4499-a095-0593e0582788" (UID: "422bc0c3-948f-4499-a095-0593e0582788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:56.648260 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.648238    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5" pod "422bc0c3-948f-4499-a095-0593e0582788" (UID: "422bc0c3-948f-4499-a095-0593e0582788"). InnerVolumeSpecName "kube-api-access-7jcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:56.748283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.748260    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jcz5\" (UniqueName: \"kubernetes.io/projected/422bc0c3-948f-4499-a095-0593e0582788-kube-api-access-7jcz5\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:56.748283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.748284    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422bc0c3-948f-4499-a095-0593e0582788-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:56.748456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:56.748293    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422bc0c3-948f-4499-a095-0593e0582788-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.419835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:57.419803    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"
Oct 30 20:39:57.419835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:57.419829    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j" event={"ID":"422bc0c3-948f-4499-a095-0593e0582788","Type":"ContainerDied","Data":"5821a5e72abaef48866bf442cb03eca78e6e9ef5b887d91b9d1d5f9bd18b7df7"}
Oct 30 20:39:57.420040 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:57.419869    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5821a5e72abaef48866bf442cb03eca78e6e9ef5b887d91b9d1d5f9bd18b7df7"
Oct 30 20:39:58.607285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.607207    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"]
Oct 30 20:39:58.608042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.608021    5967 cpu_manager.go:414] "containerMap: removing container" podUID="422bc0c3-948f-4499-a095-0593e0582788" containerName="revoke-keys"
Oct 30 20:39:58.608042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.608042    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422bc0c3-948f-4499-a095-0593e0582788" containerName="revoke-keys"
Oct 30 20:39:58.608234 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.608220    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="422bc0c3-948f-4499-a095-0593e0582788" containerName="revoke-keys"
Oct 30 20:39:58.611954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.611934    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.665878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.665850    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"]
Oct 30 20:39:58.668558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.668530    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.668694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.668567    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjcc\" (UniqueName: \"kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.668808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.668718    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.668867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.668820    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.668867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.668860    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.669011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.668975    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.710611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.710591    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"openshift-service-ca.crt\""
Oct 30 20:39:58.710726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.710695    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"default-dockercfg-wgmdp\""
Oct 30 20:39:58.712677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.712661    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"kube-root-ca.crt\""
Oct 30 20:39:58.712721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.712704    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"distinct-2-w6-4ee72d-kserve-self-signed-certs\""
Oct 30 20:39:58.770115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770080    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770161    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770242    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770280    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770316    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjcc\" (UniqueName: \"kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770457    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770479    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.770884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.770855    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.771156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.771121    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.774696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.774675    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.775043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.775022    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.828318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.828291    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjcc\" (UniqueName: \"kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc\") pod \"distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") " pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:58.928464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:58.928375    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:39:59.307864 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:39:59.307826    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b0ae2fd_4e60_48d8_9318_fb494a7594e5.slice/crio-8dfeaf85f110541ac52a2296d3ce19575f2e24f09684cdb75256e43eb4538661 WatchSource:0}: Error finding container 8dfeaf85f110541ac52a2296d3ce19575f2e24f09684cdb75256e43eb4538661: Status 404 returned error can't find the container with id 8dfeaf85f110541ac52a2296d3ce19575f2e24f09684cdb75256e43eb4538661
Oct 30 20:39:59.309132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:59.309106    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"]
Oct 30 20:39:59.435770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:59.435706    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" event={"ID":"5b0ae2fd-4e60-48d8-9318-fb494a7594e5","Type":"ContainerStarted","Data":"55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4"}
Oct 30 20:39:59.435770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:59.435771    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" event={"ID":"5b0ae2fd-4e60-48d8-9318-fb494a7594e5","Type":"ContainerStarted","Data":"8dfeaf85f110541ac52a2296d3ce19575f2e24f09684cdb75256e43eb4538661"}
Oct 30 20:39:59.554648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:59.554548    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podStartSLOduration=1.554535757 podStartE2EDuration="1.554535757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:59.550941588 +0000 UTC m=+1180.151743550" watchObservedRunningTime="2026-10-30 20:39:59.554535757 +0000 UTC m=+1180.155337717"
Oct 30 20:39:59.928610 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:39:59.928531    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:40:01.979955 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:40:01.979917    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:40:04.255564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.255530    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"]
Oct 30 20:40:04.259828 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.259813    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.312961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.312928    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:04.336911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.336877    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtc8\" (UniqueName: \"kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.337119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.337097    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.337213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.337196    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.363250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.363224    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:04.363361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.363273    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:40:04.363361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.363297    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:04.367660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.367638    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"]
Oct 30 20:40:04.438867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.438847    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.438997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.438897    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.438997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.438954    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtc8\" (UniqueName: \"kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.439625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.439602    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.441168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.441148    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.496101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.496077    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtc8\" (UniqueName: \"kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8\") pod \"maas-api-revoke-sub-b800f7b2-nsptn\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.573621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.573554    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:04.956247 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:04.956219    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2694404_ebea_48a2_b32a_67a2da8358fc.slice/crio-b687f5c243787c0402b2580b1e16856ca23857b2106434b61ea417b3558995fe WatchSource:0}: Error finding container b687f5c243787c0402b2580b1e16856ca23857b2106434b61ea417b3558995fe: Status 404 returned error can't find the container with id b687f5c243787c0402b2580b1e16856ca23857b2106434b61ea417b3558995fe
Oct 30 20:40:04.956521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:04.956499    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"]
Oct 30 20:40:05.482733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:05.482697    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn" event={"ID":"b2694404-ebea-48a2-b32a-67a2da8358fc","Type":"ContainerStarted","Data":"9f066adc0a68c072f51bb87b4fcdb7e5672b853ff6396ffc42f96b2491e1cfef"}
Oct 30 20:40:05.483120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:05.482778    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn" event={"ID":"b2694404-ebea-48a2-b32a-67a2da8358fc","Type":"ContainerStarted","Data":"b687f5c243787c0402b2580b1e16856ca23857b2106434b61ea417b3558995fe"}
Oct 30 20:40:05.598443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:05.598353    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn" podStartSLOduration=1.598338928 podStartE2EDuration="1.598338928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:05.596847343 +0000 UTC m=+1186.197649300" watchObservedRunningTime="2026-10-30 20:40:05.598338928 +0000 UTC m=+1186.199140886"
Oct 30 20:40:06.491325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:06.491295    5967 generic.go:356] "Generic (PLEG): container finished" podID="b2694404-ebea-48a2-b32a-67a2da8358fc" containerID="9f066adc0a68c072f51bb87b4fcdb7e5672b853ff6396ffc42f96b2491e1cfef" exitCode=0
Oct 30 20:40:06.491722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:06.491383    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn" event={"ID":"b2694404-ebea-48a2-b32a-67a2da8358fc","Type":"ContainerDied","Data":"9f066adc0a68c072f51bb87b4fcdb7e5672b853ff6396ffc42f96b2491e1cfef"}
Oct 30 20:40:07.654551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.654528    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:07.785700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.785619    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca\") pod \"b2694404-ebea-48a2-b32a-67a2da8358fc\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") "
Oct 30 20:40:07.785700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.785669    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8\" (UniqueName: \"kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8\") pod \"b2694404-ebea-48a2-b32a-67a2da8358fc\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") "
Oct 30 20:40:07.785948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.785770    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca\") pod \"b2694404-ebea-48a2-b32a-67a2da8358fc\" (UID: \"b2694404-ebea-48a2-b32a-67a2da8358fc\") "
Oct 30 20:40:07.786373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.786344    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca" pod "b2694404-ebea-48a2-b32a-67a2da8358fc" (UID: "b2694404-ebea-48a2-b32a-67a2da8358fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:07.788348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.788322    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8" pod "b2694404-ebea-48a2-b32a-67a2da8358fc" (UID: "b2694404-ebea-48a2-b32a-67a2da8358fc"). InnerVolumeSpecName "kube-api-access-frtc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:07.788650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.788629    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca" pod "b2694404-ebea-48a2-b32a-67a2da8358fc" (UID: "b2694404-ebea-48a2-b32a-67a2da8358fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:07.887874 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.887848    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:07.887874 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.887871    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtc8\" (UniqueName: \"kubernetes.io/projected/b2694404-ebea-48a2-b32a-67a2da8358fc-kube-api-access-frtc8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:07.888029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:07.887881    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2694404-ebea-48a2-b32a-67a2da8358fc-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:08.048306 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.048221    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"]
Oct 30 20:40:08.052783 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.052753    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d42669-8xz7n"]
Oct 30 20:40:08.505453 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.505431    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"
Oct 30 20:40:08.505679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.505470    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn" event={"ID":"b2694404-ebea-48a2-b32a-67a2da8358fc","Type":"ContainerDied","Data":"b687f5c243787c0402b2580b1e16856ca23857b2106434b61ea417b3558995fe"}
Oct 30 20:40:08.505679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.505519    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b687f5c243787c0402b2580b1e16856ca23857b2106434b61ea417b3558995fe"
Oct 30 20:40:08.929379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.929305    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:40:08.941904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:08.941879    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:40:09.113785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.113720    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"]
Oct 30 20:40:09.114319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.114306    5967 cpu_manager.go:414] "containerMap: removing container" podUID="b2694404-ebea-48a2-b32a-67a2da8358fc" containerName="revoke-keys"
Oct 30 20:40:09.114357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.114320    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2694404-ebea-48a2-b32a-67a2da8358fc" containerName="revoke-keys"
Oct 30 20:40:09.114446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.114438    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2694404-ebea-48a2-b32a-67a2da8358fc" containerName="revoke-keys"
Oct 30 20:40:09.118068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.118054    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.173119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.173039    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:09.173119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.173113    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:40:09.174667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.174647    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:09.174813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.174723    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:09.177454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.177432    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"]
Oct 30 20:40:09.201998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.201922    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.202130 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.202109    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.202295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.202279    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4sjd\" (UniqueName: \"kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.303408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.303379    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4sjd\" (UniqueName: \"kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.303607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.303435    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.303607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.303500    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.304095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.304068    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.305923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.305902    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.360328 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.360305    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4sjd\" (UniqueName: \"kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd\") pod \"maas-api-revoke-sub-7efc2280-5bhv7\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.431781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.431715    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:09.435389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.434993    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"]
Oct 30 20:40:09.439493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.439471    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.531121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.530584    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"]
Oct 30 20:40:09.531121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.530714    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:40:09.546406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.546378    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e974644-61c0-47ca-85be-1543a960ba83" path="/var/lib/kubelet/pods/5e974644-61c0-47ca-85be-1543a960ba83/volumes"
Oct 30 20:40:09.606373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606343    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.606547 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606459    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.606547 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606513    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.606725 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606700    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsn6k\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.606869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606849    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.606929 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606902    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.606998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.606976    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.607090 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.607074    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.607128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.607115    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.708575 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.708543    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.708797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.708606    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.708797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.708657    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsn6k\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.708797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.708791    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.708969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.708876    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.708969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.708936    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709017    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709043    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709063    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709138    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709296 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709252    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709431    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709548 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709498    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.709627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.709608    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.711799 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.711776    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.712029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.712006    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.764867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.764796    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.764867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.764839    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsn6k\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.779825 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.779802    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:09.818440 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:09.818412    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02c7f07_0093_40cb_8d31_b760f404af6e.slice/crio-26360bb2c79112fb45dafe2e408651f29880d196df988a639347d3446d8f3f4b WatchSource:0}: Error finding container 26360bb2c79112fb45dafe2e408651f29880d196df988a639347d3446d8f3f4b: Status 404 returned error can't find the container with id 26360bb2c79112fb45dafe2e408651f29880d196df988a639347d3446d8f3f4b
Oct 30 20:40:09.819771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:09.819718    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"]
Oct 30 20:40:10.166983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.166958    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"]
Oct 30 20:40:10.167725 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:10.167699    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1ba812_68ce_4511_b06b_ce016de4b110.slice/crio-33751758dbfd4a187de8e2e205b56a12a4dba9679da10cbbf03129227b78e688 WatchSource:0}: Error finding container 33751758dbfd4a187de8e2e205b56a12a4dba9679da10cbbf03129227b78e688: Status 404 returned error can't find the container with id 33751758dbfd4a187de8e2e205b56a12a4dba9679da10cbbf03129227b78e688
Oct 30 20:40:10.170316 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.170280    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:10.170403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.170357    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:10.170457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.170403    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:10.525097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.525063    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" event={"ID":"8e1ba812-68ce-4511-b06b-ce016de4b110","Type":"ContainerStarted","Data":"412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a"}
Oct 30 20:40:10.525284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.525124    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" event={"ID":"8e1ba812-68ce-4511-b06b-ce016de4b110","Type":"ContainerStarted","Data":"33751758dbfd4a187de8e2e205b56a12a4dba9679da10cbbf03129227b78e688"}
Oct 30 20:40:10.526941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.526918    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7" event={"ID":"f02c7f07-0093-40cb-8d31-b760f404af6e","Type":"ContainerStarted","Data":"fc80cde67dfcb63634d77330ed8f9d683ea32510e0f5d213c1c5269833d62185"}
Oct 30 20:40:10.527060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.526961    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7" event={"ID":"f02c7f07-0093-40cb-8d31-b760f404af6e","Type":"ContainerStarted","Data":"26360bb2c79112fb45dafe2e408651f29880d196df988a639347d3446d8f3f4b"}
Oct 30 20:40:10.651918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.651866    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podStartSLOduration=1.651845564 podStartE2EDuration="1.651845564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:10.65028548 +0000 UTC m=+1191.251087440" watchObservedRunningTime="2026-10-30 20:40:10.651845564 +0000 UTC m=+1191.252647525"
Oct 30 20:40:10.799851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.779972    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:10.799851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:10.782145    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7" podStartSLOduration=1.782115338 podStartE2EDuration="1.782115338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:10.780357199 +0000 UTC m=+1191.381159159" watchObservedRunningTime="2026-10-30 20:40:10.782115338 +0000 UTC m=+1191.382917299"
Oct 30 20:40:11.652488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:11.652444    5967 generic.go:356] "Generic (PLEG): container finished" podID="f02c7f07-0093-40cb-8d31-b760f404af6e" containerID="fc80cde67dfcb63634d77330ed8f9d683ea32510e0f5d213c1c5269833d62185" exitCode=0
Oct 30 20:40:11.653022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:11.652861    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7" event={"ID":"f02c7f07-0093-40cb-8d31-b760f404af6e","Type":"ContainerDied","Data":"fc80cde67dfcb63634d77330ed8f9d683ea32510e0f5d213c1c5269833d62185"}
Oct 30 20:40:11.832791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:11.816695    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:40:11.832791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:11.816980    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 20:40:12.662895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:12.662852    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:12.783552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:12.783106    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:40:12.783552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:12.783175    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 20:40:13.017911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.016866    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 30 20:40:13.092765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.090518    5967 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:40:13.092765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.090586    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:13.097762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.095535    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:40:13.231773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.228171    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:13.231773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.228392    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:13.358149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.355818    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:13.393962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.393619    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:40:13.434706 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.433977    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:40:13.559245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.559214    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:13.716831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.716802    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7" event={"ID":"f02c7f07-0093-40cb-8d31-b760f404af6e","Type":"ContainerDied","Data":"26360bb2c79112fb45dafe2e408651f29880d196df988a639347d3446d8f3f4b"}
Oct 30 20:40:13.717294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.716859    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26360bb2c79112fb45dafe2e408651f29880d196df988a639347d3446d8f3f4b"
Oct 30 20:40:13.717294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.716929    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"
Oct 30 20:40:13.748352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.748319    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd\" (UniqueName: \"kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd\") pod \"f02c7f07-0093-40cb-8d31-b760f404af6e\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") "
Oct 30 20:40:13.748779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.748757    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca\") pod \"f02c7f07-0093-40cb-8d31-b760f404af6e\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") "
Oct 30 20:40:13.748881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.748795    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca\") pod \"f02c7f07-0093-40cb-8d31-b760f404af6e\" (UID: \"f02c7f07-0093-40cb-8d31-b760f404af6e\") "
Oct 30 20:40:13.749140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.749116    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca" pod "f02c7f07-0093-40cb-8d31-b760f404af6e" (UID: "f02c7f07-0093-40cb-8d31-b760f404af6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:13.749436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.749421    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:13.751032 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.751013    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca" pod "f02c7f07-0093-40cb-8d31-b760f404af6e" (UID: "f02c7f07-0093-40cb-8d31-b760f404af6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:13.751382 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.751358    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd" pod "f02c7f07-0093-40cb-8d31-b760f404af6e" (UID: "f02c7f07-0093-40cb-8d31-b760f404af6e"). InnerVolumeSpecName "kube-api-access-p4sjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:13.791145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.791113    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:40:13.791376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.791352    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 20:40:13.850268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.850232    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4sjd\" (UniqueName: \"kubernetes.io/projected/f02c7f07-0093-40cb-8d31-b760f404af6e-kube-api-access-p4sjd\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:13.850268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:13.850264    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02c7f07-0093-40cb-8d31-b760f404af6e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:14.789767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:14.787089    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:40:14.789767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:14.787160    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 20:40:15.793905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:15.793866    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:40:15.794408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:15.793936    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 20:40:16.780629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:16.780599    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:40:16.780838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:16.780659    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 20:40:17.785029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:17.784996    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:17.785432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:17.785414    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:17.786180 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:17.786164    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:40:21.043981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.043949    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"]
Oct 30 20:40:21.044529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.044515    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f02c7f07-0093-40cb-8d31-b760f404af6e" containerName="revoke-keys"
Oct 30 20:40:21.044567 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.044529    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02c7f07-0093-40cb-8d31-b760f404af6e" containerName="revoke-keys"
Oct 30 20:40:21.044684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.044675    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02c7f07-0093-40cb-8d31-b760f404af6e" containerName="revoke-keys"
Oct 30 20:40:21.049332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.049312    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.105216 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.105189    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"]
Oct 30 20:40:21.149896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.149875    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:21.150017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.149949    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:40:21.150324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.150313    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:21.150610 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.150593    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:21.166699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.166660    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zmn\" (UniqueName: \"kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.166823 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.166702    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.166823 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.166795    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.268261 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.268234    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.268422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.268407    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zmn\" (UniqueName: \"kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.268491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.268429    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.269033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.269016    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.270699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.270682    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.325360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.325303    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zmn\" (UniqueName: \"kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn\") pod \"maas-api-revoke-sub-ca112033-rnbgx\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.363098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.363077    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:21.741217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.741185    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"]
Oct 30 20:40:21.741565 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:21.741536    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb8a9ee_df3b_493f_a32f_c8acdaaa3128.slice/crio-6f35ef3caacd4ee32b35683642140090afcfb9ee80a0fea1686b558d1a3b841c WatchSource:0}: Error finding container 6f35ef3caacd4ee32b35683642140090afcfb9ee80a0fea1686b558d1a3b841c: Status 404 returned error can't find the container with id 6f35ef3caacd4ee32b35683642140090afcfb9ee80a0fea1686b558d1a3b841c
Oct 30 20:40:21.774100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:21.774075    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx" event={"ID":"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128","Type":"ContainerStarted","Data":"6f35ef3caacd4ee32b35683642140090afcfb9ee80a0fea1686b558d1a3b841c"}
Oct 30 20:40:22.782543 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:22.782511    5967 generic.go:356] "Generic (PLEG): container finished" podID="6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" containerID="9097187f48ff5a84127db4086777f8e4cd1f44f815e1fa0e2100c5013121b658" exitCode=0
Oct 30 20:40:22.782951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:22.782594    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx" event={"ID":"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128","Type":"ContainerDied","Data":"9097187f48ff5a84127db4086777f8e4cd1f44f815e1fa0e2100c5013121b658"}
Oct 30 20:40:23.948586 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:23.948563    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:24.001021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.000993    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca\") pod \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") "
Oct 30 20:40:24.001179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.001093    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca\") pod \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") "
Oct 30 20:40:24.001179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.001148    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn\" (UniqueName: \"kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn\") pod \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\" (UID: \"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128\") "
Oct 30 20:40:24.001490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.001457    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca" pod "6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" (UID: "6cb8a9ee-df3b-493f-a32f-c8acdaaa3128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:24.003156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.003136    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca" pod "6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" (UID: "6cb8a9ee-df3b-493f-a32f-c8acdaaa3128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:24.003226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.003201    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn" pod "6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" (UID: "6cb8a9ee-df3b-493f-a32f-c8acdaaa3128"). InnerVolumeSpecName "kube-api-access-m9zmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:24.102242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.102167    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.102242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.102195    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zmn\" (UniqueName: \"kubernetes.io/projected/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-kube-api-access-m9zmn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.102242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.102208    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.799233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.799206    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"
Oct 30 20:40:24.799419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.799252    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx" event={"ID":"6cb8a9ee-df3b-493f-a32f-c8acdaaa3128","Type":"ContainerDied","Data":"6f35ef3caacd4ee32b35683642140090afcfb9ee80a0fea1686b558d1a3b841c"}
Oct 30 20:40:24.799419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:24.799304    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f35ef3caacd4ee32b35683642140090afcfb9ee80a0fea1686b558d1a3b841c"
Oct 30 20:40:25.105332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.105257    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"]
Oct 30 20:40:25.105858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.105844    5967 cpu_manager.go:414] "containerMap: removing container" podUID="6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" containerName="revoke-keys"
Oct 30 20:40:25.105924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.105859    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" containerName="revoke-keys"
Oct 30 20:40:25.105999 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.105990    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" containerName="revoke-keys"
Oct 30 20:40:25.109667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.109649    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.162862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.162838    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:40:25.163282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.163245    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:25.163433 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.163416    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:25.163541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.163527    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:25.202618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.202597    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"]
Oct 30 20:40:25.216048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.216025    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9spd\" (UniqueName: \"kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.216183 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.216159    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.216235 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.216200    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.317330 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.317303    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9spd\" (UniqueName: \"kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.317473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.317422    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.317473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.317447    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.318211 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.318182    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.319978 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.319957    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.373461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.373412    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9spd\" (UniqueName: \"kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd\") pod \"maas-api-revoke-sub-4a2d71e2-7xfpk\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.423579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.423555    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:25.798283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.798262    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"]
Oct 30 20:40:25.798841 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:25.798819    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3be2dbe_c560_41c0_9f36_516ff41b4866.slice/crio-c9ad178ae0e4b47f833e08f8f93cd065fef5365f84106a342a939d88bc93a8db WatchSource:0}: Error finding container c9ad178ae0e4b47f833e08f8f93cd065fef5365f84106a342a939d88bc93a8db: Status 404 returned error can't find the container with id c9ad178ae0e4b47f833e08f8f93cd065fef5365f84106a342a939d88bc93a8db
Oct 30 20:40:25.808619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:25.808591    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk" event={"ID":"e3be2dbe-c560-41c0-9f36-516ff41b4866","Type":"ContainerStarted","Data":"c9ad178ae0e4b47f833e08f8f93cd065fef5365f84106a342a939d88bc93a8db"}
Oct 30 20:40:26.818167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:26.818139    5967 generic.go:356] "Generic (PLEG): container finished" podID="e3be2dbe-c560-41c0-9f36-516ff41b4866" containerID="b44087b83456fce8ada2b4f93a0560708afc8d74c0423507ffe740a2ba91cc53" exitCode=0
Oct 30 20:40:26.818553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:26.818222    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk" event={"ID":"e3be2dbe-c560-41c0-9f36-516ff41b4866","Type":"ContainerDied","Data":"b44087b83456fce8ada2b4f93a0560708afc8d74c0423507ffe740a2ba91cc53"}
Oct 30 20:40:27.985420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:27.985397    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:28.053116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.053090    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd\" (UniqueName: \"kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd\") pod \"e3be2dbe-c560-41c0-9f36-516ff41b4866\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") "
Oct 30 20:40:28.053262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.053158    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca\") pod \"e3be2dbe-c560-41c0-9f36-516ff41b4866\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") "
Oct 30 20:40:28.053262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.053184    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca\") pod \"e3be2dbe-c560-41c0-9f36-516ff41b4866\" (UID: \"e3be2dbe-c560-41c0-9f36-516ff41b4866\") "
Oct 30 20:40:28.053530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.053505    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca" pod "e3be2dbe-c560-41c0-9f36-516ff41b4866" (UID: "e3be2dbe-c560-41c0-9f36-516ff41b4866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:28.053667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.053652    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:28.055172 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.055151    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd" pod "e3be2dbe-c560-41c0-9f36-516ff41b4866" (UID: "e3be2dbe-c560-41c0-9f36-516ff41b4866"). InnerVolumeSpecName "kube-api-access-n9spd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:28.055283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.055209    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca" pod "e3be2dbe-c560-41c0-9f36-516ff41b4866" (UID: "e3be2dbe-c560-41c0-9f36-516ff41b4866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:28.090508 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.090462    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"]
Oct 30 20:40:28.094119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.094100    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3d56b2-nznx4"]
Oct 30 20:40:28.155354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.155330    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2dbe-c560-41c0-9f36-516ff41b4866-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:28.155354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.155353    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9spd\" (UniqueName: \"kubernetes.io/projected/e3be2dbe-c560-41c0-9f36-516ff41b4866-kube-api-access-n9spd\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:28.834029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.834003    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"
Oct 30 20:40:28.834200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.834042    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk" event={"ID":"e3be2dbe-c560-41c0-9f36-516ff41b4866","Type":"ContainerDied","Data":"c9ad178ae0e4b47f833e08f8f93cd065fef5365f84106a342a939d88bc93a8db"}
Oct 30 20:40:28.834200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:28.834093    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ad178ae0e4b47f833e08f8f93cd065fef5365f84106a342a939d88bc93a8db"
Oct 30 20:40:29.543148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:29.543112    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acfdf56-24b9-41eb-b76a-bd3bbea1f323" path="/var/lib/kubelet/pods/8acfdf56-24b9-41eb-b76a-bd3bbea1f323/volumes"
Oct 30 20:40:31.272165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.272134    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"]
Oct 30 20:40:31.273011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.272993    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e3be2dbe-c560-41c0-9f36-516ff41b4866" containerName="revoke-keys"
Oct 30 20:40:31.273073 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.273012    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3be2dbe-c560-41c0-9f36-516ff41b4866" containerName="revoke-keys"
Oct 30 20:40:31.273226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.273100    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3be2dbe-c560-41c0-9f36-516ff41b4866" containerName="revoke-keys"
Oct 30 20:40:31.276983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.276966    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.331132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.331105    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"]
Oct 30 20:40:31.375374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.375338    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:31.375374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.375346    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:31.375573 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.375514    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:31.378113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.378094    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:40:31.394215 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.394193    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.394314 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.394244    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.394392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.394368    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzv6\" (UniqueName: \"kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.495353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.495317    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.495519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.495381    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.495519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.495420    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzv6\" (UniqueName: \"kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.495961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.495937    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.497773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.497725    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.551960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.551901    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzv6\" (UniqueName: \"kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6\") pod \"maas-api-revoke-sub-fa9062b2-dmxtb\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.592977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.592949    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:31.979545 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:40:31.979500    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:40:31.986669 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:31.986644    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0d3205_3118_4584_91e4_aabc62e29b12.slice/crio-b2fd4d33da1ca79a091b8394156b8610034d104d3fc6356274868264c152a18b WatchSource:0}: Error finding container b2fd4d33da1ca79a091b8394156b8610034d104d3fc6356274868264c152a18b: Status 404 returned error can't find the container with id b2fd4d33da1ca79a091b8394156b8610034d104d3fc6356274868264c152a18b
Oct 30 20:40:31.987249 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:31.987226    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"]
Oct 30 20:40:32.743282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.743255    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"]
Oct 30 20:40:32.747502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.747487    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:32.804325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.804295    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"]
Oct 30 20:40:32.865588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.865562    5967 generic.go:356] "Generic (PLEG): container finished" podID="0c0d3205-3118-4584-91e4-aabc62e29b12" containerID="10a942ee82bb97cfebe916bd18fa6f0d2d408be14b2535c85452e9f47b186f02" exitCode=0
Oct 30 20:40:32.865722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.865646    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb" event={"ID":"0c0d3205-3118-4584-91e4-aabc62e29b12","Type":"ContainerDied","Data":"10a942ee82bb97cfebe916bd18fa6f0d2d408be14b2535c85452e9f47b186f02"}
Oct 30 20:40:32.865722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.865698    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb" event={"ID":"0c0d3205-3118-4584-91e4-aabc62e29b12","Type":"ContainerStarted","Data":"b2fd4d33da1ca79a091b8394156b8610034d104d3fc6356274868264c152a18b"}
Oct 30 20:40:32.910733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.910688    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:32.910957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.910937    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:32.911033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:32.910995    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswvr\" (UniqueName: \"kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.012101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.012016    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.012261 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.012123    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.012261 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.012156    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswvr\" (UniqueName: \"kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.012921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.012841    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.014609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.014584    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.068707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.068671    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswvr\" (UniqueName: \"kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr\") pod \"maas-api-revoke-sub-72d6b810-mcll2\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.361281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.361192    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:33.736181 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:33.736151    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde132c84_f27e_432c_b3ff_bbfbc8877537.slice/crio-c76802ac811185766fba330d3263360c8e79605573917853ffc45efa43073c97 WatchSource:0}: Error finding container c76802ac811185766fba330d3263360c8e79605573917853ffc45efa43073c97: Status 404 returned error can't find the container with id c76802ac811185766fba330d3263360c8e79605573917853ffc45efa43073c97
Oct 30 20:40:33.736291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.736253    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"]
Oct 30 20:40:33.875411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:33.875377    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2" event={"ID":"de132c84-f27e-432c-b3ff-bbfbc8877537","Type":"ContainerStarted","Data":"c76802ac811185766fba330d3263360c8e79605573917853ffc45efa43073c97"}
Oct 30 20:40:34.028818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.028791    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:34.126867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.126839    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6\" (UniqueName: \"kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6\") pod \"0c0d3205-3118-4584-91e4-aabc62e29b12\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") "
Oct 30 20:40:34.126981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.126944    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca\") pod \"0c0d3205-3118-4584-91e4-aabc62e29b12\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") "
Oct 30 20:40:34.127035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.127020    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca\") pod \"0c0d3205-3118-4584-91e4-aabc62e29b12\" (UID: \"0c0d3205-3118-4584-91e4-aabc62e29b12\") "
Oct 30 20:40:34.127276 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.127246    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca" pod "0c0d3205-3118-4584-91e4-aabc62e29b12" (UID: "0c0d3205-3118-4584-91e4-aabc62e29b12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:34.127601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.127572    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:34.128983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.128960    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6" pod "0c0d3205-3118-4584-91e4-aabc62e29b12" (UID: "0c0d3205-3118-4584-91e4-aabc62e29b12"). InnerVolumeSpecName "kube-api-access-4bzv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:34.129062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.129014    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca" pod "0c0d3205-3118-4584-91e4-aabc62e29b12" (UID: "0c0d3205-3118-4584-91e4-aabc62e29b12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:34.229241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.229209    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d3205-3118-4584-91e4-aabc62e29b12-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:34.229241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.229242    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzv6\" (UniqueName: \"kubernetes.io/projected/0c0d3205-3118-4584-91e4-aabc62e29b12-kube-api-access-4bzv6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:34.883944 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.883875    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"
Oct 30 20:40:34.883944 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.883906    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb" event={"ID":"0c0d3205-3118-4584-91e4-aabc62e29b12","Type":"ContainerDied","Data":"b2fd4d33da1ca79a091b8394156b8610034d104d3fc6356274868264c152a18b"}
Oct 30 20:40:34.884452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.883963    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2fd4d33da1ca79a091b8394156b8610034d104d3fc6356274868264c152a18b"
Oct 30 20:40:34.885681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.885654    5967 generic.go:356] "Generic (PLEG): container finished" podID="de132c84-f27e-432c-b3ff-bbfbc8877537" containerID="47fb32ca4257446a11df93db028d292b3e3a777d97f9fe72ffcc274972ef1599" exitCode=0
Oct 30 20:40:34.885817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:34.885731    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2" event={"ID":"de132c84-f27e-432c-b3ff-bbfbc8877537","Type":"ContainerDied","Data":"47fb32ca4257446a11df93db028d292b3e3a777d97f9fe72ffcc274972ef1599"}
Oct 30 20:40:35.052689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:35.052659    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"]
Oct 30 20:40:35.056181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:35.056157    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f75dc9b-zbgwq"]
Oct 30 20:40:35.543968 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:35.543941    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceac69d8-9463-4d48-adb4-27ac3ff88122" path="/var/lib/kubelet/pods/ceac69d8-9463-4d48-adb4-27ac3ff88122/volumes"
Oct 30 20:40:36.031605 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.031585    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:36.152446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.152362    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr\" (UniqueName: \"kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr\") pod \"de132c84-f27e-432c-b3ff-bbfbc8877537\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") "
Oct 30 20:40:36.152587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.152462    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca\") pod \"de132c84-f27e-432c-b3ff-bbfbc8877537\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") "
Oct 30 20:40:36.152630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.152615    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca\") pod \"de132c84-f27e-432c-b3ff-bbfbc8877537\" (UID: \"de132c84-f27e-432c-b3ff-bbfbc8877537\") "
Oct 30 20:40:36.152906 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.152878    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca" pod "de132c84-f27e-432c-b3ff-bbfbc8877537" (UID: "de132c84-f27e-432c-b3ff-bbfbc8877537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:36.153288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.153265    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:36.154715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.154692    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca" pod "de132c84-f27e-432c-b3ff-bbfbc8877537" (UID: "de132c84-f27e-432c-b3ff-bbfbc8877537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:36.155035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.155012    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr" pod "de132c84-f27e-432c-b3ff-bbfbc8877537" (UID: "de132c84-f27e-432c-b3ff-bbfbc8877537"). InnerVolumeSpecName "kube-api-access-cswvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:36.254824 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.254801    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswvr\" (UniqueName: \"kubernetes.io/projected/de132c84-f27e-432c-b3ff-bbfbc8877537-kube-api-access-cswvr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:36.254824 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.254826    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de132c84-f27e-432c-b3ff-bbfbc8877537-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:36.903019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.902991    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"
Oct 30 20:40:36.903178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.903018    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2" event={"ID":"de132c84-f27e-432c-b3ff-bbfbc8877537","Type":"ContainerDied","Data":"c76802ac811185766fba330d3263360c8e79605573917853ffc45efa43073c97"}
Oct 30 20:40:36.903178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:36.903071    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76802ac811185766fba330d3263360c8e79605573917853ffc45efa43073c97"
Oct 30 20:40:37.471892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.471858    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"]
Oct 30 20:40:37.472462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.472448    5967 cpu_manager.go:414] "containerMap: removing container" podUID="de132c84-f27e-432c-b3ff-bbfbc8877537" containerName="revoke-keys"
Oct 30 20:40:37.472501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.472462    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de132c84-f27e-432c-b3ff-bbfbc8877537" containerName="revoke-keys"
Oct 30 20:40:37.472534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.472500    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0c0d3205-3118-4584-91e4-aabc62e29b12" containerName="revoke-keys"
Oct 30 20:40:37.472534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.472505    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0d3205-3118-4584-91e4-aabc62e29b12" containerName="revoke-keys"
Oct 30 20:40:37.472609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.472600    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0d3205-3118-4584-91e4-aabc62e29b12" containerName="revoke-keys"
Oct 30 20:40:37.472644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.472615    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="de132c84-f27e-432c-b3ff-bbfbc8877537" containerName="revoke-keys"
Oct 30 20:40:37.476225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.476208    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.521117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.521091    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"]
Oct 30 20:40:37.525204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.525189    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570044 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570015    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570063    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570079    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570243 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570219    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570275    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570314    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570347    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570370    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570384    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v927v\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570399    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570551    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570581    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570647 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570635    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570655    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570789    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570861    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxz7d\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.570896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570891    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.570987 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.570918    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.579292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.579270    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"]
Oct 30 20:40:37.671665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671613    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxz7d\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.671665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671662    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671692    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671777    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671835    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671857    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671915    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671967    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.671994    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672042    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672084    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672108    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v927v\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672132    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672201    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672225    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672261    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672284    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672362    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672517    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672821    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.672941    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.673059    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.673628    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.673904    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.674130    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.674203    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.674702    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.674845    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.675791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.675527    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.677800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.677754    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.677931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.677917    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.678321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.678299    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.679660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.679636    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"]
Oct 30 20:40:37.730692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.730626    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.730855 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.730735    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxz7d\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.730911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.730874    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v927v\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.752717 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.752696    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:37.790139 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.790117    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:37.847873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:37.847847    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:38.048019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.047896    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"]
Oct 30 20:40:38.052346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.052319    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82bab324-xbvtw"]
Oct 30 20:40:38.224164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.224107    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"]
Oct 30 20:40:38.259414 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:38.259385    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185df700_2f7c_4ff9_af4e_66a44b5bca2b.slice/crio-6fbe4bb00b6f66ea829f7881ff67d85cb4c53559d2dbb5ce2f2ff68708764bc2 WatchSource:0}: Error finding container 6fbe4bb00b6f66ea829f7881ff67d85cb4c53559d2dbb5ce2f2ff68708764bc2: Status 404 returned error can't find the container with id 6fbe4bb00b6f66ea829f7881ff67d85cb4c53559d2dbb5ce2f2ff68708764bc2
Oct 30 20:40:38.262025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.261994    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.262123 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.262058    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.262123 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.262088    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.276020 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:38.275994    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a87637_d9da_44bc_af84_44750f82613b.slice/crio-bec9caefbade56c5700c57bdb58fe1798511b2767b7c3bfccd53bbfe6ec73d5f WatchSource:0}: Error finding container bec9caefbade56c5700c57bdb58fe1798511b2767b7c3bfccd53bbfe6ec73d5f: Status 404 returned error can't find the container with id bec9caefbade56c5700c57bdb58fe1798511b2767b7c3bfccd53bbfe6ec73d5f
Oct 30 20:40:38.276869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.276849    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"]
Oct 30 20:40:38.278268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.278236    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.278663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.278612    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.278880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.278670    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:40:38.922425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.922379    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" event={"ID":"185df700-2f7c-4ff9-af4e-66a44b5bca2b","Type":"ContainerStarted","Data":"02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a"}
Oct 30 20:40:38.922987 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.922453    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" event={"ID":"185df700-2f7c-4ff9-af4e-66a44b5bca2b","Type":"ContainerStarted","Data":"6fbe4bb00b6f66ea829f7881ff67d85cb4c53559d2dbb5ce2f2ff68708764bc2"}
Oct 30 20:40:38.924174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.924150    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" event={"ID":"97a87637-d9da-44bc-af84-44750f82613b","Type":"ContainerStarted","Data":"7541845f5641ea7a9980cc7c5c90375c589f57e0f79136ed89a63a6a9c0d107f"}
Oct 30 20:40:38.924307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:38.924193    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" event={"ID":"97a87637-d9da-44bc-af84-44750f82613b","Type":"ContainerStarted","Data":"bec9caefbade56c5700c57bdb58fe1798511b2767b7c3bfccd53bbfe6ec73d5f"}
Oct 30 20:40:39.043531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.043478    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podStartSLOduration=2.043465231 podStartE2EDuration="2.043465231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:39.039130593 +0000 UTC m=+1219.639932552" watchObservedRunningTime="2026-10-30 20:40:39.043465231 +0000 UTC m=+1219.644267190"
Oct 30 20:40:39.163760 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.162131    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podStartSLOduration=2.16211167 podStartE2EDuration="2.16211167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:39.155636049 +0000 UTC m=+1219.756438008" watchObservedRunningTime="2026-10-30 20:40:39.16211167 +0000 UTC m=+1219.762913630"
Oct 30 20:40:39.544564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.544532    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01cbffef-dfb1-4388-beb0-60f97c2ebb47" path="/var/lib/kubelet/pods/01cbffef-dfb1-4388-beb0-60f97c2ebb47/volumes"
Oct 30 20:40:39.799859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.792660    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:39.799859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.792877    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 20:40:39.799859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.792921    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 20:40:39.848460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.848432    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:39.849725 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.849692    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:40:39.849872 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:39.849758    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 20:40:40.794766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:40.794107    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 20:40:40.794766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:40.794167    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 20:40:40.849055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:40.849019    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:40:40.849204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:40.849068    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 20:40:41.803765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:41.802888    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 20:40:41.803765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:41.802967    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 20:40:41.848981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:41.848944    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:40:41.849144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:41.848997    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 20:40:42.804772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:42.800127    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.804772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:42.800188    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 20:40:42.849009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:42.848978    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.849169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:42.849032    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 20:40:43.802203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:43.802162    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.802498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:43.802469    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 20:40:43.849110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:43.849076    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.849491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:43.849135    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 20:40:44.791262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:44.791225    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.791443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:44.791291    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 20:40:44.849336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:44.849304    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.849714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:44.849358    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 20:40:45.053309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.053223    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"]
Oct 30 20:40:45.067021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.066989    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"]
Oct 30 20:40:45.100728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.100689    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaaa0c64-9sqrx"]
Oct 30 20:40:45.104513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.104485    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41880891-kz9x7"]
Oct 30 20:40:45.550080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.550038    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41025734-6e53-4794-9ea0-d3498fec8071" path="/var/lib/kubelet/pods/41025734-6e53-4794-9ea0-d3498fec8071/volumes"
Oct 30 20:40:45.550772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.550713    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58ebaf7-fe37-41d5-974a-4aca6074810f" path="/var/lib/kubelet/pods/f58ebaf7-fe37-41d5-974a-4aca6074810f/volumes"
Oct 30 20:40:45.724885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.724842    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"]
Oct 30 20:40:45.731909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.731885    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.788370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.788330    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"]
Oct 30 20:40:45.829012 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.828936    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:45.829012 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.828957    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:40:45.829224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.829060    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:45.837681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.837638    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:45.848947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.848921    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.849136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.849118    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.849216 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.849144    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqnn\" (UniqueName: \"kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.852955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.852934    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:45.853332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.853228    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:45.853915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.853899    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:40:45.950710 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.950462    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.950710 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.950507    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqnn\" (UniqueName: \"kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.950710 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.950686    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.951342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.951323    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.953451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.953434    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:45.958071 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.958048    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:45.958445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.958389    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:45.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:45.959350    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:40:46.006452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:46.006432    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqnn\" (UniqueName: \"kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn\") pod \"maas-api-revoke-sub-e44ba5ff-7s9r2\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:46.054179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:46.054152    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:46.445336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:46.445314    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"]
Oct 30 20:40:46.445866 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:46.445843    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b746798_4a6f_473d_b0e5_3fc86018b23e.slice/crio-50c38179dce2e5b16ab0723195519842ebe0ce3ba8c181adb38253f129e05595 WatchSource:0}: Error finding container 50c38179dce2e5b16ab0723195519842ebe0ce3ba8c181adb38253f129e05595: Status 404 returned error can't find the container with id 50c38179dce2e5b16ab0723195519842ebe0ce3ba8c181adb38253f129e05595
Oct 30 20:40:47.038976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:47.038817    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2" event={"ID":"1b746798-4a6f-473d-b0e5-3fc86018b23e","Type":"ContainerStarted","Data":"1a42d076d4339bcf6c478fa171200a54b540554fdd0a0b6437ddf5c6808fc08c"}
Oct 30 20:40:47.038976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:47.038883    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2" event={"ID":"1b746798-4a6f-473d-b0e5-3fc86018b23e","Type":"ContainerStarted","Data":"50c38179dce2e5b16ab0723195519842ebe0ce3ba8c181adb38253f129e05595"}
Oct 30 20:40:47.158785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:47.158714    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2" podStartSLOduration=2.158701437 podStartE2EDuration="2.158701437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:47.155608464 +0000 UTC m=+1227.756410424" watchObservedRunningTime="2026-10-30 20:40:47.158701437 +0000 UTC m=+1227.759503445"
Oct 30 20:40:48.047909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:48.047879    5967 generic.go:356] "Generic (PLEG): container finished" podID="1b746798-4a6f-473d-b0e5-3fc86018b23e" containerID="1a42d076d4339bcf6c478fa171200a54b540554fdd0a0b6437ddf5c6808fc08c" exitCode=0
Oct 30 20:40:48.048340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:48.047959    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2" event={"ID":"1b746798-4a6f-473d-b0e5-3fc86018b23e","Type":"ContainerDied","Data":"1a42d076d4339bcf6c478fa171200a54b540554fdd0a0b6437ddf5c6808fc08c"}
Oct 30 20:40:49.391493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.391113    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:49.529763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.529606    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca\") pod \"1b746798-4a6f-473d-b0e5-3fc86018b23e\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") "
Oct 30 20:40:49.529763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.529716    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn\" (UniqueName: \"kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn\") pod \"1b746798-4a6f-473d-b0e5-3fc86018b23e\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") "
Oct 30 20:40:49.530039 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.529796    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca\") pod \"1b746798-4a6f-473d-b0e5-3fc86018b23e\" (UID: \"1b746798-4a6f-473d-b0e5-3fc86018b23e\") "
Oct 30 20:40:49.542765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.530784    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca" pod "1b746798-4a6f-473d-b0e5-3fc86018b23e" (UID: "1b746798-4a6f-473d-b0e5-3fc86018b23e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:49.543253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.543226    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca" pod "1b746798-4a6f-473d-b0e5-3fc86018b23e" (UID: "1b746798-4a6f-473d-b0e5-3fc86018b23e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:49.666034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.665944    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:49.666257 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.666240    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b746798-4a6f-473d-b0e5-3fc86018b23e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:49.666612 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.666591    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn" pod "1b746798-4a6f-473d-b0e5-3fc86018b23e" (UID: "1b746798-4a6f-473d-b0e5-3fc86018b23e"). InnerVolumeSpecName "kube-api-access-ggqnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:49.767213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.767176    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"]
Oct 30 20:40:49.768201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.768174    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1b746798-4a6f-473d-b0e5-3fc86018b23e" containerName="revoke-keys"
Oct 30 20:40:49.768201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.768202    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b746798-4a6f-473d-b0e5-3fc86018b23e" containerName="revoke-keys"
Oct 30 20:40:49.768374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.768361    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b746798-4a6f-473d-b0e5-3fc86018b23e" containerName="revoke-keys"
Oct 30 20:40:49.795765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.769781    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqnn\" (UniqueName: \"kubernetes.io/projected/1b746798-4a6f-473d-b0e5-3fc86018b23e-kube-api-access-ggqnn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:49.861032 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.856508    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"]
Oct 30 20:40:49.861032 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.856638    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.878326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.874945    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.878326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.874992    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ln2\" (UniqueName: \"kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.878326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.875044    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.999767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.995575    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.999767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.995629    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ln2\" (UniqueName: \"kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.999767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.995682    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:49.999767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:49.996912    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:50.030597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.030570    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:50.056423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.056393    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ln2\" (UniqueName: \"kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2\") pod \"maas-api-revoke-sub-a8c80311-66g7n\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:50.148484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.148445    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2" event={"ID":"1b746798-4a6f-473d-b0e5-3fc86018b23e","Type":"ContainerDied","Data":"50c38179dce2e5b16ab0723195519842ebe0ce3ba8c181adb38253f129e05595"}
Oct 30 20:40:50.148667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.148530    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c38179dce2e5b16ab0723195519842ebe0ce3ba8c181adb38253f129e05595"
Oct 30 20:40:50.148667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.148617    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"
Oct 30 20:40:50.183764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.181891    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:50.305761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.301857    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"]
Oct 30 20:40:50.531811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.525662    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"]
Oct 30 20:40:50.531811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.525814    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.665896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.661678    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.708943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.708533    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.708943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.708771    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.837140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.836713    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.837319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.837149    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.837723 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.837697    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.837881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.837860    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.846314 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.846283    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.894073 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.893948    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz\") pod \"maas-api-revoke-sub-0e3257db-zplmq\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.912835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.903078    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:50.988660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:50.988353    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"]
Oct 30 20:40:51.164166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:51.164127    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n" event={"ID":"d183b1af-5f73-45ad-907f-1e3fd2f60ee9","Type":"ContainerStarted","Data":"068c0f8f94fea416a938770e8554d61bcf57ab64ff4cd6e41fc12ead50915f1d"}
Oct 30 20:40:51.300679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:51.300649    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"]
Oct 30 20:40:51.301123 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:51.301098    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49a6f89_4b5f_4fb1_bbcb_b40f448a27e4.slice/crio-d16c998345c798b6a4a99f1b0b0f6cd5d0731506967dab6e9de2fde720d3ae23 WatchSource:0}: Error finding container d16c998345c798b6a4a99f1b0b0f6cd5d0731506967dab6e9de2fde720d3ae23: Status 404 returned error can't find the container with id d16c998345c798b6a4a99f1b0b0f6cd5d0731506967dab6e9de2fde720d3ae23
Oct 30 20:40:52.174871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:52.174836    5967 generic.go:356] "Generic (PLEG): container finished" podID="d183b1af-5f73-45ad-907f-1e3fd2f60ee9" containerID="b79cbe1db7b7fdf8791786e8e262eef1161015fd5cdcbff40eb2c516d6b34a13" exitCode=0
Oct 30 20:40:52.175303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:52.174973    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n" event={"ID":"d183b1af-5f73-45ad-907f-1e3fd2f60ee9","Type":"ContainerDied","Data":"b79cbe1db7b7fdf8791786e8e262eef1161015fd5cdcbff40eb2c516d6b34a13"}
Oct 30 20:40:52.176918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:52.176896    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq" event={"ID":"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4","Type":"ContainerStarted","Data":"4df9b65067dffedf2ca06ccde7caa09d24111af11d0df1c4601dfe94c82c5607"}
Oct 30 20:40:52.177046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:52.176932    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq" event={"ID":"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4","Type":"ContainerStarted","Data":"d16c998345c798b6a4a99f1b0b0f6cd5d0731506967dab6e9de2fde720d3ae23"}
Oct 30 20:40:52.407915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:52.407868    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq" podStartSLOduration=2.4078528869999998 podStartE2EDuration="2.407852887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:52.40429924 +0000 UTC m=+1233.005101199" watchObservedRunningTime="2026-10-30 20:40:52.407852887 +0000 UTC m=+1233.008654847"
Oct 30 20:40:53.185873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.185841    5967 generic.go:356] "Generic (PLEG): container finished" podID="f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" containerID="4df9b65067dffedf2ca06ccde7caa09d24111af11d0df1c4601dfe94c82c5607" exitCode=0
Oct 30 20:40:53.186350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.185938    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq" event={"ID":"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4","Type":"ContainerDied","Data":"4df9b65067dffedf2ca06ccde7caa09d24111af11d0df1c4601dfe94c82c5607"}
Oct 30 20:40:53.331829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.331804    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:53.390506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.390473    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca\") pod \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") "
Oct 30 20:40:53.390660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.390511    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2\" (UniqueName: \"kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2\") pod \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") "
Oct 30 20:40:53.390719 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.390696    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca\") pod \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\" (UID: \"d183b1af-5f73-45ad-907f-1e3fd2f60ee9\") "
Oct 30 20:40:53.391046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.391018    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca" pod "d183b1af-5f73-45ad-907f-1e3fd2f60ee9" (UID: "d183b1af-5f73-45ad-907f-1e3fd2f60ee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:53.391586 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.391561    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:53.392701 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.392680    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca" pod "d183b1af-5f73-45ad-907f-1e3fd2f60ee9" (UID: "d183b1af-5f73-45ad-907f-1e3fd2f60ee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:53.392809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.392701    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2" pod "d183b1af-5f73-45ad-907f-1e3fd2f60ee9" (UID: "d183b1af-5f73-45ad-907f-1e3fd2f60ee9"). InnerVolumeSpecName "kube-api-access-42ln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:53.493049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.493016    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:53.493049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.493047    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ln2\" (UniqueName: \"kubernetes.io/projected/d183b1af-5f73-45ad-907f-1e3fd2f60ee9-kube-api-access-42ln2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:53.915881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.915801    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"]
Oct 30 20:40:53.916476 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.916457    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d183b1af-5f73-45ad-907f-1e3fd2f60ee9" containerName="revoke-keys"
Oct 30 20:40:53.916579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.916478    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d183b1af-5f73-45ad-907f-1e3fd2f60ee9" containerName="revoke-keys"
Oct 30 20:40:53.916698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.916684    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d183b1af-5f73-45ad-907f-1e3fd2f60ee9" containerName="revoke-keys"
Oct 30 20:40:53.920549 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.920524    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:53.978285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.978256    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"]
Oct 30 20:40:53.998111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.998085    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:53.998255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.998143    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:53.998388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:53.998364    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfbkm\" (UniqueName: \"kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.049928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.049900    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"]
Oct 30 20:40:54.088951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.088919    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000fe0f5-5lgvs"]
Oct 30 20:40:54.100109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.100078    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfbkm\" (UniqueName: \"kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.100259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.100169    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.100259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.100199    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.100813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.100789    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.102616 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.102594    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.156378 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.156348    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfbkm\" (UniqueName: \"kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm\") pod \"maas-api-revoke-sub-7db6080c-tvj5p\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.196181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.196110    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"
Oct 30 20:40:54.196181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.196139    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n" event={"ID":"d183b1af-5f73-45ad-907f-1e3fd2f60ee9","Type":"ContainerDied","Data":"068c0f8f94fea416a938770e8554d61bcf57ab64ff4cd6e41fc12ead50915f1d"}
Oct 30 20:40:54.196657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.196182    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="068c0f8f94fea416a938770e8554d61bcf57ab64ff4cd6e41fc12ead50915f1d"
Oct 30 20:40:54.233841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.233814    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:54.374728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.374705    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:54.403401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.403381    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz\") pod \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") "
Oct 30 20:40:54.403529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.403422    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca\") pod \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") "
Oct 30 20:40:54.403529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.403497    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca\") pod \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\" (UID: \"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4\") "
Oct 30 20:40:54.403837 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.403813    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca" pod "f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" (UID: "f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:54.403944 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.403924    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:54.405468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.405448    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca" pod "f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" (UID: "f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:54.405659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.405632    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz" pod "f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" (UID: "f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4"). InnerVolumeSpecName "kube-api-access-vmgwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:54.505125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.505091    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:54.505125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.505118    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4-kube-api-access-vmgwz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:54.533521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.533495    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"]
Oct 30 20:40:54.534421 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.534402    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" containerName="revoke-keys"
Oct 30 20:40:54.534482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.534425    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" containerName="revoke-keys"
Oct 30 20:40:54.534617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.534605    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" containerName="revoke-keys"
Oct 30 20:40:54.539036 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.539011    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.596620 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.596583    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"]
Oct 30 20:40:54.606151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.606125    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.606318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.606236    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.606381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.606322    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvvgk\" (UniqueName: \"kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.630240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.630213    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"]
Oct 30 20:40:54.631505 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:54.631483    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589cb23b_c231_4ee3_a715_80f021c382c2.slice/crio-6af14691c36a2fc4b4ed18df268809ff939907ce7a1db03bcaa42f451e261e96 WatchSource:0}: Error finding container 6af14691c36a2fc4b4ed18df268809ff939907ce7a1db03bcaa42f451e261e96: Status 404 returned error can't find the container with id 6af14691c36a2fc4b4ed18df268809ff939907ce7a1db03bcaa42f451e261e96
Oct 30 20:40:54.707851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.707813    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvvgk\" (UniqueName: \"kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.708038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.707942    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.708038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.707999    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.708552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.708534    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.710322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.710303    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.765494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.765423    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvvgk\" (UniqueName: \"kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk\") pod \"maas-api-revoke-sub-c779eb29-ptzk4\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:54.863106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:54.863074    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:55.205316 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.205226    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p" event={"ID":"589cb23b-c231-4ee3-a715-80f021c382c2","Type":"ContainerStarted","Data":"77d9c9c71936cec57dedacf6852a5492bb36914013cd28afbe478e2d94390e3e"}
Oct 30 20:40:55.205316 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.205287    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p" event={"ID":"589cb23b-c231-4ee3-a715-80f021c382c2","Type":"ContainerStarted","Data":"6af14691c36a2fc4b4ed18df268809ff939907ce7a1db03bcaa42f451e261e96"}
Oct 30 20:40:55.206786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.206762    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"
Oct 30 20:40:55.206786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.206775    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq" event={"ID":"f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4","Type":"ContainerDied","Data":"d16c998345c798b6a4a99f1b0b0f6cd5d0731506967dab6e9de2fde720d3ae23"}
Oct 30 20:40:55.206928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.206815    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16c998345c798b6a4a99f1b0b0f6cd5d0731506967dab6e9de2fde720d3ae23"
Oct 30 20:40:55.245171 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:55.245148    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cfc00f_a341_4d43_84eb_fff343200ef9.slice/crio-dde40c4e2f27f066946d706526303c839535aba4c826c7578073f45a6625506e WatchSource:0}: Error finding container dde40c4e2f27f066946d706526303c839535aba4c826c7578073f45a6625506e: Status 404 returned error can't find the container with id dde40c4e2f27f066946d706526303c839535aba4c826c7578073f45a6625506e
Oct 30 20:40:55.245552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.245533    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"]
Oct 30 20:40:55.321415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.321361    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p" podStartSLOduration=2.321343504 podStartE2EDuration="2.321343504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:55.320235406 +0000 UTC m=+1235.921037365" watchObservedRunningTime="2026-10-30 20:40:55.321343504 +0000 UTC m=+1235.922145466"
Oct 30 20:40:55.543122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.543091    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0430ee17-c021-4a92-8835-93d40ad2db2a" path="/var/lib/kubelet/pods/0430ee17-c021-4a92-8835-93d40ad2db2a/volumes"
Oct 30 20:40:55.649572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.649541    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"]
Oct 30 20:40:55.653853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.653837    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.713502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.713455    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"]
Oct 30 20:40:55.722269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.722241    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257rq\" (UniqueName: \"kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.722402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.722312    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.722560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.722541    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.823457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.823391    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.823457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.823450    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257rq\" (UniqueName: \"kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.823664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.823483    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.824085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.824060    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.825780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.825757    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.880426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.880407    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257rq\" (UniqueName: \"kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq\") pod \"maas-api-revoke-sub-000a85ca-r9tdc\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:55.969696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:55.969596    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:56.233064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:56.233030    5967 generic.go:356] "Generic (PLEG): container finished" podID="589cb23b-c231-4ee3-a715-80f021c382c2" containerID="77d9c9c71936cec57dedacf6852a5492bb36914013cd28afbe478e2d94390e3e" exitCode=0
Oct 30 20:40:56.233490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:56.233310    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p" event={"ID":"589cb23b-c231-4ee3-a715-80f021c382c2","Type":"ContainerDied","Data":"77d9c9c71936cec57dedacf6852a5492bb36914013cd28afbe478e2d94390e3e"}
Oct 30 20:40:56.237243 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:56.235533    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4" event={"ID":"e8cfc00f-a341-4d43-84eb-fff343200ef9","Type":"ContainerStarted","Data":"4bc7674a60d04917a565cac604b18d2b68cf46e6bfed8e59b5cdf736f107b83b"}
Oct 30 20:40:56.237243 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:56.235592    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4" event={"ID":"e8cfc00f-a341-4d43-84eb-fff343200ef9","Type":"ContainerStarted","Data":"dde40c4e2f27f066946d706526303c839535aba4c826c7578073f45a6625506e"}
Oct 30 20:40:56.355106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:56.355079    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"]
Oct 30 20:40:56.392616 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:56.392587    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3414b8_48e9_4417_865e_66209cb49a7b.slice/crio-e85f173545dd51b2a8529966f2dfee31f559d2262e18be107fa5c4eb004cd971 WatchSource:0}: Error finding container e85f173545dd51b2a8529966f2dfee31f559d2262e18be107fa5c4eb004cd971: Status 404 returned error can't find the container with id e85f173545dd51b2a8529966f2dfee31f559d2262e18be107fa5c4eb004cd971
Oct 30 20:40:56.461581 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:56.461532    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4" podStartSLOduration=2.461520304 podStartE2EDuration="2.461520304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:56.460176816 +0000 UTC m=+1237.060978776" watchObservedRunningTime="2026-10-30 20:40:56.461520304 +0000 UTC m=+1237.062322276"
Oct 30 20:40:57.247279 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.247247    5967 generic.go:356] "Generic (PLEG): container finished" podID="e8cfc00f-a341-4d43-84eb-fff343200ef9" containerID="4bc7674a60d04917a565cac604b18d2b68cf46e6bfed8e59b5cdf736f107b83b" exitCode=0
Oct 30 20:40:57.247660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.247323    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4" event={"ID":"e8cfc00f-a341-4d43-84eb-fff343200ef9","Type":"ContainerDied","Data":"4bc7674a60d04917a565cac604b18d2b68cf46e6bfed8e59b5cdf736f107b83b"}
Oct 30 20:40:57.249422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.249393    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc" event={"ID":"dc3414b8-48e9-4417-865e-66209cb49a7b","Type":"ContainerStarted","Data":"17e8dd575441237a255ac3205e5ca01778252a4f66c410e03e9ae617e2029001"}
Oct 30 20:40:57.249562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.249441    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc" event={"ID":"dc3414b8-48e9-4417-865e-66209cb49a7b","Type":"ContainerStarted","Data":"e85f173545dd51b2a8529966f2dfee31f559d2262e18be107fa5c4eb004cd971"}
Oct 30 20:40:57.403915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.403889    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:57.449190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.449163    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca\") pod \"589cb23b-c231-4ee3-a715-80f021c382c2\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") "
Oct 30 20:40:57.449340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.449206    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca\") pod \"589cb23b-c231-4ee3-a715-80f021c382c2\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") "
Oct 30 20:40:57.449340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.449252    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm\" (UniqueName: \"kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm\") pod \"589cb23b-c231-4ee3-a715-80f021c382c2\" (UID: \"589cb23b-c231-4ee3-a715-80f021c382c2\") "
Oct 30 20:40:57.449532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.449506    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca" pod "589cb23b-c231-4ee3-a715-80f021c382c2" (UID: "589cb23b-c231-4ee3-a715-80f021c382c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:57.449854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.449837    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:57.451297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.451281    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca" pod "589cb23b-c231-4ee3-a715-80f021c382c2" (UID: "589cb23b-c231-4ee3-a715-80f021c382c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:57.451442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.451428    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm" pod "589cb23b-c231-4ee3-a715-80f021c382c2" (UID: "589cb23b-c231-4ee3-a715-80f021c382c2"). InnerVolumeSpecName "kube-api-access-bfbkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:57.550617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.550551    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589cb23b-c231-4ee3-a715-80f021c382c2-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:57.550782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:57.550668    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfbkm\" (UniqueName: \"kubernetes.io/projected/589cb23b-c231-4ee3-a715-80f021c382c2-kube-api-access-bfbkm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.047214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.047187    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"]
Oct 30 20:40:58.050921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.050896    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1886feb4-j9kxj"]
Oct 30 20:40:58.138898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.138869    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"]
Oct 30 20:40:58.139510 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.139497    5967 cpu_manager.go:414] "containerMap: removing container" podUID="589cb23b-c231-4ee3-a715-80f021c382c2" containerName="revoke-keys"
Oct 30 20:40:58.139558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.139516    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589cb23b-c231-4ee3-a715-80f021c382c2" containerName="revoke-keys"
Oct 30 20:40:58.139619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.139610    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="589cb23b-c231-4ee3-a715-80f021c382c2" containerName="revoke-keys"
Oct 30 20:40:58.143322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.143307    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.201505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.201467    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"]
Oct 30 20:40:58.258112 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.258077    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.258576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.258270    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.258576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.258359    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8sj\" (UniqueName: \"kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.259315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.259288    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"
Oct 30 20:40:58.259475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.259309    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p" event={"ID":"589cb23b-c231-4ee3-a715-80f021c382c2","Type":"ContainerDied","Data":"6af14691c36a2fc4b4ed18df268809ff939907ce7a1db03bcaa42f451e261e96"}
Oct 30 20:40:58.259475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.259362    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6af14691c36a2fc4b4ed18df268809ff939907ce7a1db03bcaa42f451e261e96"
Oct 30 20:40:58.261231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.261204    5967 generic.go:356] "Generic (PLEG): container finished" podID="dc3414b8-48e9-4417-865e-66209cb49a7b" containerID="17e8dd575441237a255ac3205e5ca01778252a4f66c410e03e9ae617e2029001" exitCode=0
Oct 30 20:40:58.261387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.261240    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc" event={"ID":"dc3414b8-48e9-4417-865e-66209cb49a7b","Type":"ContainerDied","Data":"17e8dd575441237a255ac3205e5ca01778252a4f66c410e03e9ae617e2029001"}
Oct 30 20:40:58.371462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.365537    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8sj\" (UniqueName: \"kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.371462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.365662    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.371462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.367474    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.371462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.368167    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.371462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.370754    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.422625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.422591    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8sj\" (UniqueName: \"kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj\") pod \"maas-api-revoke-sub-c41545d2-5tfl5\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.448584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.448561    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:58.451028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.451012    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:58.456331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.456313    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.569053    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq\" (UniqueName: \"kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq\") pod \"dc3414b8-48e9-4417-865e-66209cb49a7b\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") "
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.569172    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca\") pod \"e8cfc00f-a341-4d43-84eb-fff343200ef9\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") "
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.569276    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca\") pod \"dc3414b8-48e9-4417-865e-66209cb49a7b\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") "
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.569311    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk\" (UniqueName: \"kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk\") pod \"e8cfc00f-a341-4d43-84eb-fff343200ef9\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") "
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.569344    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca\") pod \"e8cfc00f-a341-4d43-84eb-fff343200ef9\" (UID: \"e8cfc00f-a341-4d43-84eb-fff343200ef9\") "
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.569412    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca\") pod \"dc3414b8-48e9-4417-865e-66209cb49a7b\" (UID: \"dc3414b8-48e9-4417-865e-66209cb49a7b\") "
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.570546    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca" pod "e8cfc00f-a341-4d43-84eb-fff343200ef9" (UID: "e8cfc00f-a341-4d43-84eb-fff343200ef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:58.570713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.570628    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca" pod "dc3414b8-48e9-4417-865e-66209cb49a7b" (UID: "dc3414b8-48e9-4417-865e-66209cb49a7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:58.575205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.575173    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca" pod "e8cfc00f-a341-4d43-84eb-fff343200ef9" (UID: "e8cfc00f-a341-4d43-84eb-fff343200ef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:58.575205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.575185    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca" pod "dc3414b8-48e9-4417-865e-66209cb49a7b" (UID: "dc3414b8-48e9-4417-865e-66209cb49a7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:58.575352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.575222    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk" pod "e8cfc00f-a341-4d43-84eb-fff343200ef9" (UID: "e8cfc00f-a341-4d43-84eb-fff343200ef9"). InnerVolumeSpecName "kube-api-access-vvvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:58.578617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.578599    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq" pod "dc3414b8-48e9-4417-865e-66209cb49a7b" (UID: "dc3414b8-48e9-4417-865e-66209cb49a7b"). InnerVolumeSpecName "kube-api-access-257rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:58.670848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.670826    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.670848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.670846    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257rq\" (UniqueName: \"kubernetes.io/projected/dc3414b8-48e9-4417-865e-66209cb49a7b-kube-api-access-257rq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.671037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.670855    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.671037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.670866    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3414b8-48e9-4417-865e-66209cb49a7b-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.671037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.670874    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvvgk\" (UniqueName: \"kubernetes.io/projected/e8cfc00f-a341-4d43-84eb-fff343200ef9-kube-api-access-vvvgk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.671037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.670883    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cfc00f-a341-4d43-84eb-fff343200ef9-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:58.837841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:58.837804    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"]
Oct 30 20:40:58.838165 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:58.838143    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e13fb3_a54b_43fd_8474_c570a30c9e02.slice/crio-0e1f1fd4a5e4f14411d8828a23e1bc9d5c817e258c55031315bf47191ec7a07e WatchSource:0}: Error finding container 0e1f1fd4a5e4f14411d8828a23e1bc9d5c817e258c55031315bf47191ec7a07e: Status 404 returned error can't find the container with id 0e1f1fd4a5e4f14411d8828a23e1bc9d5c817e258c55031315bf47191ec7a07e
Oct 30 20:40:59.025047 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025015    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"]
Oct 30 20:40:59.025707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025692    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e8cfc00f-a341-4d43-84eb-fff343200ef9" containerName="revoke-keys"
Oct 30 20:40:59.025707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025708    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cfc00f-a341-4d43-84eb-fff343200ef9" containerName="revoke-keys"
Oct 30 20:40:59.025867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025748    5967 cpu_manager.go:414] "containerMap: removing container" podUID="dc3414b8-48e9-4417-865e-66209cb49a7b" containerName="revoke-keys"
Oct 30 20:40:59.025867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025755    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3414b8-48e9-4417-865e-66209cb49a7b" containerName="revoke-keys"
Oct 30 20:40:59.025867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025857    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3414b8-48e9-4417-865e-66209cb49a7b" containerName="revoke-keys"
Oct 30 20:40:59.026024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.025872    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cfc00f-a341-4d43-84eb-fff343200ef9" containerName="revoke-keys"
Oct 30 20:40:59.030780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.030760    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.075194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.075171    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.075363 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.075344    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.075454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.075440    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fb94\" (UniqueName: \"kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.086772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.086695    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"]
Oct 30 20:40:59.176825 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.176802    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.176952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.176880    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fb94\" (UniqueName: \"kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.177002 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.176954    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.177444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.177423    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.179286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.179268    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.241069 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.241048    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fb94\" (UniqueName: \"kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94\") pod \"maas-api-revoke-sub-cffb967e-ns9rq\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.270118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.270093    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"
Oct 30 20:40:59.270500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.270099    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4" event={"ID":"e8cfc00f-a341-4d43-84eb-fff343200ef9","Type":"ContainerDied","Data":"dde40c4e2f27f066946d706526303c839535aba4c826c7578073f45a6625506e"}
Oct 30 20:40:59.270500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.270204    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde40c4e2f27f066946d706526303c839535aba4c826c7578073f45a6625506e"
Oct 30 20:40:59.271702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.271680    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"
Oct 30 20:40:59.271835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.271727    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc" event={"ID":"dc3414b8-48e9-4417-865e-66209cb49a7b","Type":"ContainerDied","Data":"e85f173545dd51b2a8529966f2dfee31f559d2262e18be107fa5c4eb004cd971"}
Oct 30 20:40:59.271835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.271793    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85f173545dd51b2a8529966f2dfee31f559d2262e18be107fa5c4eb004cd971"
Oct 30 20:40:59.273636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.273601    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5" event={"ID":"e0e13fb3-a54b-43fd-8474-c570a30c9e02","Type":"ContainerStarted","Data":"fbbe7aa7a4f78addcf8a9585bf0f20f95a3d053d0cd0ddb616ea1600488e2869"}
Oct 30 20:40:59.273762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.273661    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5" event={"ID":"e0e13fb3-a54b-43fd-8474-c570a30c9e02","Type":"ContainerStarted","Data":"0e1f1fd4a5e4f14411d8828a23e1bc9d5c817e258c55031315bf47191ec7a07e"}
Oct 30 20:40:59.343922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.343837    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:40:59.400911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.400844    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5" podStartSLOduration=1.400825851 podStartE2EDuration="1.400825851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:59.398388451 +0000 UTC m=+1239.999190410" watchObservedRunningTime="2026-10-30 20:40:59.400825851 +0000 UTC m=+1240.001627812"
Oct 30 20:40:59.544375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.544347    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8e6ff5-56ee-401a-9354-89486b6822bd" path="/var/lib/kubelet/pods/fe8e6ff5-56ee-401a-9354-89486b6822bd/volumes"
Oct 30 20:40:59.727115 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:40:59.727086    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765516f0_185e_4e1a_a7df_5b374dccc564.slice/crio-9cde4a29d04eed5ce2289594a7102052808e475546a8c9156c00f2c66d32de85 WatchSource:0}: Error finding container 9cde4a29d04eed5ce2289594a7102052808e475546a8c9156c00f2c66d32de85: Status 404 returned error can't find the container with id 9cde4a29d04eed5ce2289594a7102052808e475546a8c9156c00f2c66d32de85
Oct 30 20:40:59.728696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:40:59.728601    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"]
Oct 30 20:41:00.052949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.052920    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"]
Oct 30 20:41:00.056630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.056607    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318093ab-j4r2r"]
Oct 30 20:41:00.283643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.283552    5967 generic.go:356] "Generic (PLEG): container finished" podID="e0e13fb3-a54b-43fd-8474-c570a30c9e02" containerID="fbbe7aa7a4f78addcf8a9585bf0f20f95a3d053d0cd0ddb616ea1600488e2869" exitCode=0
Oct 30 20:41:00.283643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.283586    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5" event={"ID":"e0e13fb3-a54b-43fd-8474-c570a30c9e02","Type":"ContainerDied","Data":"fbbe7aa7a4f78addcf8a9585bf0f20f95a3d053d0cd0ddb616ea1600488e2869"}
Oct 30 20:41:00.285713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.285688    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq" event={"ID":"765516f0-185e-4e1a-a7df-5b374dccc564","Type":"ContainerStarted","Data":"9f1ef5bef7f2f35fb9514392f6b79ed071e8da33e1fe2e9a3b5ebe81acb376d6"}
Oct 30 20:41:00.285874 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.285734    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq" event={"ID":"765516f0-185e-4e1a-a7df-5b374dccc564","Type":"ContainerStarted","Data":"9cde4a29d04eed5ce2289594a7102052808e475546a8c9156c00f2c66d32de85"}
Oct 30 20:41:00.478205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.478174    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"]
Oct 30 20:41:00.482728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.482708    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.519632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.519591    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq" podStartSLOduration=2.51957964 podStartE2EDuration="2.51957964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:00.515565864 +0000 UTC m=+1241.116367825" watchObservedRunningTime="2026-10-30 20:41:00.51957964 +0000 UTC m=+1241.120381784"
Oct 30 20:41:00.541472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.541404    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"]
Oct 30 20:41:00.592129 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.592099    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.592129 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.592130    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.592355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.592276    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.693766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.693728    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.693886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.693775    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.693886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.693824    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.694533 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.694445    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.696071 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.696049    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.751475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.751448    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq\") pod \"maas-api-revoke-sub-156ebb11-xjpns\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:00.796359 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:00.796295    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:01.186886 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:01.186856    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb3cb89_f75c_48aa_b308_bdf782704ae2.slice/crio-0ea367e7c7f182c2681426ff148cbac9332606a75d542ba5818e4adaea61280a WatchSource:0}: Error finding container 0ea367e7c7f182c2681426ff148cbac9332606a75d542ba5818e4adaea61280a: Status 404 returned error can't find the container with id 0ea367e7c7f182c2681426ff148cbac9332606a75d542ba5818e4adaea61280a
Oct 30 20:41:01.189025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.188996    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"]
Oct 30 20:41:01.296792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.296723    5967 generic.go:356] "Generic (PLEG): container finished" podID="765516f0-185e-4e1a-a7df-5b374dccc564" containerID="9f1ef5bef7f2f35fb9514392f6b79ed071e8da33e1fe2e9a3b5ebe81acb376d6" exitCode=0
Oct 30 20:41:01.297212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.296851    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq" event={"ID":"765516f0-185e-4e1a-a7df-5b374dccc564","Type":"ContainerDied","Data":"9f1ef5bef7f2f35fb9514392f6b79ed071e8da33e1fe2e9a3b5ebe81acb376d6"}
Oct 30 20:41:01.298373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.298347    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns" event={"ID":"bfb3cb89-f75c-48aa-b308-bdf782704ae2","Type":"ContainerStarted","Data":"0ea367e7c7f182c2681426ff148cbac9332606a75d542ba5818e4adaea61280a"}
Oct 30 20:41:01.449306 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.449279    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:41:01.507230 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.507191    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj\" (UniqueName: \"kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj\") pod \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") "
Oct 30 20:41:01.507418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.507283    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca\") pod \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") "
Oct 30 20:41:01.507418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.507332    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca\") pod \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\" (UID: \"e0e13fb3-a54b-43fd-8474-c570a30c9e02\") "
Oct 30 20:41:01.507713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.507690    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca" pod "e0e13fb3-a54b-43fd-8474-c570a30c9e02" (UID: "e0e13fb3-a54b-43fd-8474-c570a30c9e02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:01.508083 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.508058    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:01.509664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.509630    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca" pod "e0e13fb3-a54b-43fd-8474-c570a30c9e02" (UID: "e0e13fb3-a54b-43fd-8474-c570a30c9e02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:01.509664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.509650    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj" pod "e0e13fb3-a54b-43fd-8474-c570a30c9e02" (UID: "e0e13fb3-a54b-43fd-8474-c570a30c9e02"). InnerVolumeSpecName "kube-api-access-nl8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:01.545889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.545857    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f7835c-ee10-4439-82ef-9554af6ec202" path="/var/lib/kubelet/pods/f3f7835c-ee10-4439-82ef-9554af6ec202/volumes"
Oct 30 20:41:01.609384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.609358    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e13fb3-a54b-43fd-8474-c570a30c9e02-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:01.609384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:01.609386    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8sj\" (UniqueName: \"kubernetes.io/projected/e0e13fb3-a54b-43fd-8474-c570a30c9e02-kube-api-access-nl8sj\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:01.985829 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:41:01.985793    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:41:02.057895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.057863    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"]
Oct 30 20:41:02.062892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.062862    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee8d684-lwmsn"]
Oct 30 20:41:02.313291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.313205    5967 generic.go:356] "Generic (PLEG): container finished" podID="bfb3cb89-f75c-48aa-b308-bdf782704ae2" containerID="bae2f9738e8e0bbb79bdaabd9f63b958e8e6870ba57e8a36a74ef7af1e19c5be" exitCode=0
Oct 30 20:41:02.313729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.313283    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns" event={"ID":"bfb3cb89-f75c-48aa-b308-bdf782704ae2","Type":"ContainerDied","Data":"bae2f9738e8e0bbb79bdaabd9f63b958e8e6870ba57e8a36a74ef7af1e19c5be"}
Oct 30 20:41:02.314963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.314942    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"
Oct 30 20:41:02.315086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.314982    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5" event={"ID":"e0e13fb3-a54b-43fd-8474-c570a30c9e02","Type":"ContainerDied","Data":"0e1f1fd4a5e4f14411d8828a23e1bc9d5c817e258c55031315bf47191ec7a07e"}
Oct 30 20:41:02.315086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.315022    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1f1fd4a5e4f14411d8828a23e1bc9d5c817e258c55031315bf47191ec7a07e"
Oct 30 20:41:02.540465 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.540437    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:41:02.625095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.625015    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca\") pod \"765516f0-185e-4e1a-a7df-5b374dccc564\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") "
Oct 30 20:41:02.625095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.625088    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca\") pod \"765516f0-185e-4e1a-a7df-5b374dccc564\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") "
Oct 30 20:41:02.625341 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.625205    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94\" (UniqueName: \"kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94\") pod \"765516f0-185e-4e1a-a7df-5b374dccc564\" (UID: \"765516f0-185e-4e1a-a7df-5b374dccc564\") "
Oct 30 20:41:02.625432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.625400    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca" pod "765516f0-185e-4e1a-a7df-5b374dccc564" (UID: "765516f0-185e-4e1a-a7df-5b374dccc564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:02.626815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.626707    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:02.631777 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.627726    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94" pod "765516f0-185e-4e1a-a7df-5b374dccc564" (UID: "765516f0-185e-4e1a-a7df-5b374dccc564"). InnerVolumeSpecName "kube-api-access-8fb94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:02.631777 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.628095    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca" pod "765516f0-185e-4e1a-a7df-5b374dccc564" (UID: "765516f0-185e-4e1a-a7df-5b374dccc564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:02.727931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.727893    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765516f0-185e-4e1a-a7df-5b374dccc564-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:02.727931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:02.727928    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fb94\" (UniqueName: \"kubernetes.io/projected/765516f0-185e-4e1a-a7df-5b374dccc564-kube-api-access-8fb94\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:03.056092 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.056046    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"]
Oct 30 20:41:03.062339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.062314    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0409d895-7dwzj"]
Oct 30 20:41:03.327988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.327894    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq" event={"ID":"765516f0-185e-4e1a-a7df-5b374dccc564","Type":"ContainerDied","Data":"9cde4a29d04eed5ce2289594a7102052808e475546a8c9156c00f2c66d32de85"}
Oct 30 20:41:03.329128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.328986    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cde4a29d04eed5ce2289594a7102052808e475546a8c9156c00f2c66d32de85"
Oct 30 20:41:03.329128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.329055    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"
Oct 30 20:41:03.535505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.535475    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:03.553809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.552487    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47798542-aa0e-48bf-a15f-b25de925551a" path="/var/lib/kubelet/pods/47798542-aa0e-48bf-a15f-b25de925551a/volumes"
Oct 30 20:41:03.553809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.553162    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bba0197-e717-42b1-8a57-0594d43b0f38" path="/var/lib/kubelet/pods/6bba0197-e717-42b1-8a57-0594d43b0f38/volumes"
Oct 30 20:41:03.651481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.651143    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq\") pod \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") "
Oct 30 20:41:03.655831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.652310    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca\") pod \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") "
Oct 30 20:41:03.655831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.652372    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca\") pod \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\" (UID: \"bfb3cb89-f75c-48aa-b308-bdf782704ae2\") "
Oct 30 20:41:03.657587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.657559    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca" pod "bfb3cb89-f75c-48aa-b308-bdf782704ae2" (UID: "bfb3cb89-f75c-48aa-b308-bdf782704ae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:03.663419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.663379    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq" pod "bfb3cb89-f75c-48aa-b308-bdf782704ae2" (UID: "bfb3cb89-f75c-48aa-b308-bdf782704ae2"). InnerVolumeSpecName "kube-api-access-g82bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:03.664722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.664700    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca" pod "bfb3cb89-f75c-48aa-b308-bdf782704ae2" (UID: "bfb3cb89-f75c-48aa-b308-bdf782704ae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:03.754137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.754102    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:03.754137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.754131    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb3cb89-f75c-48aa-b308-bdf782704ae2-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:03.754137 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:03.754143    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82bq\" (UniqueName: \"kubernetes.io/projected/bfb3cb89-f75c-48aa-b308-bdf782704ae2-kube-api-access-g82bq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:04.337070 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:04.337046    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"
Oct 30 20:41:04.337070 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:04.337062    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns" event={"ID":"bfb3cb89-f75c-48aa-b308-bdf782704ae2","Type":"ContainerDied","Data":"0ea367e7c7f182c2681426ff148cbac9332606a75d542ba5818e4adaea61280a"}
Oct 30 20:41:04.337578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:04.337109    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ea367e7c7f182c2681426ff148cbac9332606a75d542ba5818e4adaea61280a"
Oct 30 20:41:06.607010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.606975    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"]
Oct 30 20:41:06.607991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.607971    5967 cpu_manager.go:414] "containerMap: removing container" podUID="765516f0-185e-4e1a-a7df-5b374dccc564" containerName="revoke-keys"
Oct 30 20:41:06.608053 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.607994    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765516f0-185e-4e1a-a7df-5b374dccc564" containerName="revoke-keys"
Oct 30 20:41:06.608053 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608013    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e0e13fb3-a54b-43fd-8474-c570a30c9e02" containerName="revoke-keys"
Oct 30 20:41:06.608053 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608021    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e13fb3-a54b-43fd-8474-c570a30c9e02" containerName="revoke-keys"
Oct 30 20:41:06.608151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608068    5967 cpu_manager.go:414] "containerMap: removing container" podUID="bfb3cb89-f75c-48aa-b308-bdf782704ae2" containerName="revoke-keys"
Oct 30 20:41:06.608151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608078    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb3cb89-f75c-48aa-b308-bdf782704ae2" containerName="revoke-keys"
Oct 30 20:41:06.608276 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608264    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="765516f0-185e-4e1a-a7df-5b374dccc564" containerName="revoke-keys"
Oct 30 20:41:06.608317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608282    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e13fb3-a54b-43fd-8474-c570a30c9e02" containerName="revoke-keys"
Oct 30 20:41:06.608317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.608300    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb3cb89-f75c-48aa-b308-bdf782704ae2" containerName="revoke-keys"
Oct 30 20:41:06.615819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.613790    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.669637 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.669602    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"]
Oct 30 20:41:06.713836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.713799    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:41:06.713836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.713834    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:41:06.714026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.713843    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:41:06.719202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.719173    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:06.742128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.742099    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.742343 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.742325    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.742421 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.742380    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9w9s\" (UniqueName: \"kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.843561 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.843534    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.843665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.843584    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9w9s\" (UniqueName: \"kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.843665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.843632    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.844215 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.844195    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.846353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.846333    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.900945 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.900893    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9w9s\" (UniqueName: \"kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s\") pod \"maas-api-revoke-sub-e190cd35-mvpsw\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:06.937287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:06.937258    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:07.316662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:07.316628    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"]
Oct 30 20:41:07.316797 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:07.316714    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47b7017_aa1e_4fae_a439_5b0f5d43d664.slice/crio-99d7a3795848f2200b18c69e266233c865a258d83cfe36702dec4b2cf89eb576 WatchSource:0}: Error finding container 99d7a3795848f2200b18c69e266233c865a258d83cfe36702dec4b2cf89eb576: Status 404 returned error can't find the container with id 99d7a3795848f2200b18c69e266233c865a258d83cfe36702dec4b2cf89eb576
Oct 30 20:41:07.370436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:07.370410    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw" event={"ID":"e47b7017-aa1e-4fae-a439-5b0f5d43d664","Type":"ContainerStarted","Data":"99d7a3795848f2200b18c69e266233c865a258d83cfe36702dec4b2cf89eb576"}
Oct 30 20:41:07.895719 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:07.895693    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"]
Oct 30 20:41:07.899971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:07.899952    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:07.958765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:07.958681    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"]
Oct 30 20:41:08.059490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.059457    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.059614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.059526    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.059699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.059676    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5g4\" (UniqueName: \"kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.160657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.160623    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5g4\" (UniqueName: \"kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.160848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.160704    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.160848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.160780    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.161378 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.161359    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.163017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.162992    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.218703 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.218681    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5g4\" (UniqueName: \"kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4\") pod \"maas-api-revoke-sub-fbfe83f4-2z4lh\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.381052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.381022    5967 generic.go:356] "Generic (PLEG): container finished" podID="e47b7017-aa1e-4fae-a439-5b0f5d43d664" containerID="fabe892216ab397e17cb8d33fd675c3322bb28c69bd413278380457a3b6ba865" exitCode=0
Oct 30 20:41:08.381250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.381114    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw" event={"ID":"e47b7017-aa1e-4fae-a439-5b0f5d43d664","Type":"ContainerDied","Data":"fabe892216ab397e17cb8d33fd675c3322bb28c69bd413278380457a3b6ba865"}
Oct 30 20:41:08.514390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.514318    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:08.892940 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:08.892902    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cd695e_2b63_45e2_bd97_ec9c487f6abd.slice/crio-0c6d47e3161df0390bb11a365c984a056d514165ff16d38a4e1c31aa8fbe8b48 WatchSource:0}: Error finding container 0c6d47e3161df0390bb11a365c984a056d514165ff16d38a4e1c31aa8fbe8b48: Status 404 returned error can't find the container with id 0c6d47e3161df0390bb11a365c984a056d514165ff16d38a4e1c31aa8fbe8b48
Oct 30 20:41:08.894123 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:08.893694    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"]
Oct 30 20:41:09.049281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.049250    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"]
Oct 30 20:41:09.054344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.054321    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f4e4f-prbfv"]
Oct 30 20:41:09.392567 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.392533    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh" event={"ID":"82cd695e-2b63-45e2-bd97-ec9c487f6abd","Type":"ContainerStarted","Data":"b8545228cc14ab34a44d8794ed0cc69af615d606c121a22b5f0b0e22c5493c15"}
Oct 30 20:41:09.392761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.392593    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh" event={"ID":"82cd695e-2b63-45e2-bd97-ec9c487f6abd","Type":"ContainerStarted","Data":"0c6d47e3161df0390bb11a365c984a056d514165ff16d38a4e1c31aa8fbe8b48"}
Oct 30 20:41:09.510597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.510551    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh" podStartSLOduration=2.510536817 podStartE2EDuration="2.510536817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:09.507314627 +0000 UTC m=+1250.108116587" watchObservedRunningTime="2026-10-30 20:41:09.510536817 +0000 UTC m=+1250.111338776"
Oct 30 20:41:09.545022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.544996    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff37fd42-38e1-4d90-a833-cce6ae42b829" path="/var/lib/kubelet/pods/ff37fd42-38e1-4d90-a833-cce6ae42b829/volumes"
Oct 30 20:41:09.555392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.555376    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:09.680481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.680452    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s\" (UniqueName: \"kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s\") pod \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") "
Oct 30 20:41:09.680658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.680490    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca\") pod \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") "
Oct 30 20:41:09.680658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.680626    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca\") pod \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\" (UID: \"e47b7017-aa1e-4fae-a439-5b0f5d43d664\") "
Oct 30 20:41:09.681037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.681002    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca" pod "e47b7017-aa1e-4fae-a439-5b0f5d43d664" (UID: "e47b7017-aa1e-4fae-a439-5b0f5d43d664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:09.681584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.681562    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:09.682911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.682892    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s" pod "e47b7017-aa1e-4fae-a439-5b0f5d43d664" (UID: "e47b7017-aa1e-4fae-a439-5b0f5d43d664"). InnerVolumeSpecName "kube-api-access-z9w9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:09.682974 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.682916    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca" pod "e47b7017-aa1e-4fae-a439-5b0f5d43d664" (UID: "e47b7017-aa1e-4fae-a439-5b0f5d43d664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:09.783341 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.783315    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9w9s\" (UniqueName: \"kubernetes.io/projected/e47b7017-aa1e-4fae-a439-5b0f5d43d664-kube-api-access-z9w9s\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:09.783341 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:09.783343    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47b7017-aa1e-4fae-a439-5b0f5d43d664-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:10.076060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.075964    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"]
Oct 30 20:41:10.080784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.080752    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db988f65-djch6"]
Oct 30 20:41:10.401557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.401477    5967 generic.go:356] "Generic (PLEG): container finished" podID="82cd695e-2b63-45e2-bd97-ec9c487f6abd" containerID="b8545228cc14ab34a44d8794ed0cc69af615d606c121a22b5f0b0e22c5493c15" exitCode=0
Oct 30 20:41:10.401715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.401555    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh" event={"ID":"82cd695e-2b63-45e2-bd97-ec9c487f6abd","Type":"ContainerDied","Data":"b8545228cc14ab34a44d8794ed0cc69af615d606c121a22b5f0b0e22c5493c15"}
Oct 30 20:41:10.403078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.403052    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw" event={"ID":"e47b7017-aa1e-4fae-a439-5b0f5d43d664","Type":"ContainerDied","Data":"99d7a3795848f2200b18c69e266233c865a258d83cfe36702dec4b2cf89eb576"}
Oct 30 20:41:10.403078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.403076    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"
Oct 30 20:41:10.403247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:10.403098    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d7a3795848f2200b18c69e266233c865a258d83cfe36702dec4b2cf89eb576"
Oct 30 20:41:11.544167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.544136    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff" path="/var/lib/kubelet/pods/3ecea26b-71cf-46c5-84e9-3bc3ee7f5dff/volumes"
Oct 30 20:41:11.578533 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.578511    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:11.704997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.704926    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca\") pod \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") "
Oct 30 20:41:11.705158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.705036    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca\") pod \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") "
Oct 30 20:41:11.705282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.705263    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4\" (UniqueName: \"kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4\") pod \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\" (UID: \"82cd695e-2b63-45e2-bd97-ec9c487f6abd\") "
Oct 30 20:41:11.705340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.705302    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca" pod "82cd695e-2b63-45e2-bd97-ec9c487f6abd" (UID: "82cd695e-2b63-45e2-bd97-ec9c487f6abd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:11.705793 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.705729    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:11.707495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.707473    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca" pod "82cd695e-2b63-45e2-bd97-ec9c487f6abd" (UID: "82cd695e-2b63-45e2-bd97-ec9c487f6abd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:11.707716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.707699    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4" pod "82cd695e-2b63-45e2-bd97-ec9c487f6abd" (UID: "82cd695e-2b63-45e2-bd97-ec9c487f6abd"). InnerVolumeSpecName "kube-api-access-fn5g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:11.806518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.806491    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5g4\" (UniqueName: \"kubernetes.io/projected/82cd695e-2b63-45e2-bd97-ec9c487f6abd-kube-api-access-fn5g4\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:11.806518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:11.806515    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd695e-2b63-45e2-bd97-ec9c487f6abd-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:12.421188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:12.421163    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"
Oct 30 20:41:12.421344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:12.421198    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh" event={"ID":"82cd695e-2b63-45e2-bd97-ec9c487f6abd","Type":"ContainerDied","Data":"0c6d47e3161df0390bb11a365c984a056d514165ff16d38a4e1c31aa8fbe8b48"}
Oct 30 20:41:12.421344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:12.421275    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6d47e3161df0390bb11a365c984a056d514165ff16d38a4e1c31aa8fbe8b48"
Oct 30 20:41:14.101426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.101396    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"]
Oct 30 20:41:14.102062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.102046    5967 cpu_manager.go:414] "containerMap: removing container" podUID="82cd695e-2b63-45e2-bd97-ec9c487f6abd" containerName="revoke-keys"
Oct 30 20:41:14.102062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.102063    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cd695e-2b63-45e2-bd97-ec9c487f6abd" containerName="revoke-keys"
Oct 30 20:41:14.102141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.102091    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e47b7017-aa1e-4fae-a439-5b0f5d43d664" containerName="revoke-keys"
Oct 30 20:41:14.102141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.102096    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47b7017-aa1e-4fae-a439-5b0f5d43d664" containerName="revoke-keys"
Oct 30 20:41:14.102209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.102199    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cd695e-2b63-45e2-bd97-ec9c487f6abd" containerName="revoke-keys"
Oct 30 20:41:14.102255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.102211    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47b7017-aa1e-4fae-a439-5b0f5d43d664" containerName="revoke-keys"
Oct 30 20:41:14.105945 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.105924    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.207416 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.207386    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:41:14.207416 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.207414    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:41:14.207671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.207386    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:14.207864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.207842    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:41:14.212457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.212434    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"]
Oct 30 20:41:14.236434 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.236407    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.236585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.236570    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.236715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.236687    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmg5\" (UniqueName: \"kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.338569 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.338536    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.338707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.338616    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmg5\" (UniqueName: \"kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.338707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.338667    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.339283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.339264    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.340998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.340981    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.396508 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.396450    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmg5\" (UniqueName: \"kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5\") pod \"maas-api-revoke-sub-68525d2e-4fzfn\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.421486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.421468    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:14.807405 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:14.807375    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db77f87_7883_4939_980c_6d38a4fa91cd.slice/crio-238beb961a6bf09950a7b9446e46978897c33be77707ea1a95cc39fbec276f2b WatchSource:0}: Error finding container 238beb961a6bf09950a7b9446e46978897c33be77707ea1a95cc39fbec276f2b: Status 404 returned error can't find the container with id 238beb961a6bf09950a7b9446e46978897c33be77707ea1a95cc39fbec276f2b
Oct 30 20:41:14.807405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:14.807405    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"]
Oct 30 20:41:15.153402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.153355    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"]
Oct 30 20:41:15.157690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.157673    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.215023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.214994    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"]
Oct 30 20:41:15.252530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.252490    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5l4\" (UniqueName: \"kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.252846 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.252779    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.252846 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.252821    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.354272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.354196    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.354272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.354244    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.354454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.354331    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5l4\" (UniqueName: \"kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.354955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.354894    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.356779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.356715    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.411321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.411289    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5l4\" (UniqueName: \"kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4\") pod \"maas-api-revoke-sub-0b5f3a1e-g4gjd\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.453054 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.453021    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn" event={"ID":"1db77f87-7883-4939-980c-6d38a4fa91cd","Type":"ContainerStarted","Data":"f1eb1c51381a231108c2f05c9a32ef54503754ecc16b90814d65cc22d1c2b387"}
Oct 30 20:41:15.453241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.453085    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn" event={"ID":"1db77f87-7883-4939-980c-6d38a4fa91cd","Type":"ContainerStarted","Data":"238beb961a6bf09950a7b9446e46978897c33be77707ea1a95cc39fbec276f2b"}
Oct 30 20:41:15.470893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.470856    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:15.575132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.575082    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn" podStartSLOduration=1.5750691369999998 podStartE2EDuration="1.575069137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:15.5739369 +0000 UTC m=+1256.174738860" watchObservedRunningTime="2026-10-30 20:41:15.575069137 +0000 UTC m=+1256.175871096"
Oct 30 20:41:15.856114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:15.855413    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"]
Oct 30 20:41:15.858178 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:15.858136    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0553767a_9895_4946_b66c_516203b38291.slice/crio-92f8337c64e5fbc866c4a1f07ec90b4a87d2779b44c4e11539886634e853d235 WatchSource:0}: Error finding container 92f8337c64e5fbc866c4a1f07ec90b4a87d2779b44c4e11539886634e853d235: Status 404 returned error can't find the container with id 92f8337c64e5fbc866c4a1f07ec90b4a87d2779b44c4e11539886634e853d235
Oct 30 20:41:16.430085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.430010    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"]
Oct 30 20:41:16.434233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.434218    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.463310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.463286    5967 generic.go:356] "Generic (PLEG): container finished" podID="1db77f87-7883-4939-980c-6d38a4fa91cd" containerID="f1eb1c51381a231108c2f05c9a32ef54503754ecc16b90814d65cc22d1c2b387" exitCode=0
Oct 30 20:41:16.463416 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.463375    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn" event={"ID":"1db77f87-7883-4939-980c-6d38a4fa91cd","Type":"ContainerDied","Data":"f1eb1c51381a231108c2f05c9a32ef54503754ecc16b90814d65cc22d1c2b387"}
Oct 30 20:41:16.465040 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.465021    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd" event={"ID":"0553767a-9895-4946-b66c-516203b38291","Type":"ContainerStarted","Data":"63da20a0608ecffd7b023baa587c061d3e1781c6d0d079ef351e16f07f8b8e4c"}
Oct 30 20:41:16.465132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.465063    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd" event={"ID":"0553767a-9895-4946-b66c-516203b38291","Type":"ContainerStarted","Data":"92f8337c64e5fbc866c4a1f07ec90b4a87d2779b44c4e11539886634e853d235"}
Oct 30 20:41:16.493083 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.493059    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"]
Oct 30 20:41:16.570167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.570132    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjm6\" (UniqueName: \"kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.570332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.570303    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.570476 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.570461    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.670789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.670726    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd" podStartSLOduration=1.670713291 podStartE2EDuration="1.670713291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:16.666265094 +0000 UTC m=+1257.267067054" watchObservedRunningTime="2026-10-30 20:41:16.670713291 +0000 UTC m=+1257.271515272"
Oct 30 20:41:16.672068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.672046    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.672210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.672192    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.672301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.672282    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjm6\" (UniqueName: \"kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.672867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.672850    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.675068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.675047    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.729289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.729263    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjm6\" (UniqueName: \"kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6\") pod \"maas-api-revoke-sub-f5804185-8kvgq\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:16.748717 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:16.748690    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:17.132463 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:17.132434    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1937f6d0_aa8f_4061_8e22_9291b1746c50.slice/crio-acacccac7bee95e1d245e7f2cb43b3dd130ef26e601bb3bebdc0d19618bd9439 WatchSource:0}: Error finding container acacccac7bee95e1d245e7f2cb43b3dd130ef26e601bb3bebdc0d19618bd9439: Status 404 returned error can't find the container with id acacccac7bee95e1d245e7f2cb43b3dd130ef26e601bb3bebdc0d19618bd9439
Oct 30 20:41:17.132639 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.132481    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"]
Oct 30 20:41:17.404178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.404152    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"]
Oct 30 20:41:17.409902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.409883    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.467326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.467303    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"]
Oct 30 20:41:17.478149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.478122    5967 generic.go:356] "Generic (PLEG): container finished" podID="0553767a-9895-4946-b66c-516203b38291" containerID="63da20a0608ecffd7b023baa587c061d3e1781c6d0d079ef351e16f07f8b8e4c" exitCode=0
Oct 30 20:41:17.478293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.478228    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd" event={"ID":"0553767a-9895-4946-b66c-516203b38291","Type":"ContainerDied","Data":"63da20a0608ecffd7b023baa587c061d3e1781c6d0d079ef351e16f07f8b8e4c"}
Oct 30 20:41:17.479853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.479829    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq" event={"ID":"1937f6d0-aa8f-4061-8e22-9291b1746c50","Type":"ContainerStarted","Data":"acacccac7bee95e1d245e7f2cb43b3dd130ef26e601bb3bebdc0d19618bd9439"}
Oct 30 20:41:17.482196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.482168    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.482291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.482256    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.482357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.482341    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlthn\" (UniqueName: \"kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.585317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.583376    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.585317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.583432    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.585317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.583481    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlthn\" (UniqueName: \"kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.585317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.584331    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.594180 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.591781    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.645005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.644975    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlthn\" (UniqueName: \"kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn\") pod \"maas-api-revoke-sub-7f5f06e1-2rl5r\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.669619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.669597    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:17.684089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.684067    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5\" (UniqueName: \"kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5\") pod \"1db77f87-7883-4939-980c-6d38a4fa91cd\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") "
Oct 30 20:41:17.684214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.684115    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca\") pod \"1db77f87-7883-4939-980c-6d38a4fa91cd\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") "
Oct 30 20:41:17.684288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.684214    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca\") pod \"1db77f87-7883-4939-980c-6d38a4fa91cd\" (UID: \"1db77f87-7883-4939-980c-6d38a4fa91cd\") "
Oct 30 20:41:17.684496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.684471    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca" pod "1db77f87-7883-4939-980c-6d38a4fa91cd" (UID: "1db77f87-7883-4939-980c-6d38a4fa91cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:17.684722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.684705    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:17.686272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.686247    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5" pod "1db77f87-7883-4939-980c-6d38a4fa91cd" (UID: "1db77f87-7883-4939-980c-6d38a4fa91cd"). InnerVolumeSpecName "kube-api-access-2vmg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:17.686340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.686309    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca" pod "1db77f87-7883-4939-980c-6d38a4fa91cd" (UID: "1db77f87-7883-4939-980c-6d38a4fa91cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:17.723302 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.723274    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:17.789969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.789762    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vmg5\" (UniqueName: \"kubernetes.io/projected/1db77f87-7883-4939-980c-6d38a4fa91cd-kube-api-access-2vmg5\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:17.789969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:17.789799    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db77f87-7883-4939-980c-6d38a4fa91cd-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.099031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.099003    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"]
Oct 30 20:41:18.099841 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:18.099818    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010a8f71_8fcf_4c29_88a7_5887f1f08f92.slice/crio-e7648515e252bde2a4ee630c00ba6b8a99ac6100419aa0acaac57b79e59a45a1 WatchSource:0}: Error finding container e7648515e252bde2a4ee630c00ba6b8a99ac6100419aa0acaac57b79e59a45a1: Status 404 returned error can't find the container with id e7648515e252bde2a4ee630c00ba6b8a99ac6100419aa0acaac57b79e59a45a1
Oct 30 20:41:18.489050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.489025    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"
Oct 30 20:41:18.489463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.489066    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn" event={"ID":"1db77f87-7883-4939-980c-6d38a4fa91cd","Type":"ContainerDied","Data":"238beb961a6bf09950a7b9446e46978897c33be77707ea1a95cc39fbec276f2b"}
Oct 30 20:41:18.489463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.489111    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="238beb961a6bf09950a7b9446e46978897c33be77707ea1a95cc39fbec276f2b"
Oct 30 20:41:18.490940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.490912    5967 generic.go:356] "Generic (PLEG): container finished" podID="1937f6d0-aa8f-4061-8e22-9291b1746c50" containerID="002775af1fde945f245ec8062df99df60639ece20ad4cf774a561c4646c529d4" exitCode=0
Oct 30 20:41:18.491070 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.490955    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq" event={"ID":"1937f6d0-aa8f-4061-8e22-9291b1746c50","Type":"ContainerDied","Data":"002775af1fde945f245ec8062df99df60639ece20ad4cf774a561c4646c529d4"}
Oct 30 20:41:18.493168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.493010    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r" event={"ID":"010a8f71-8fcf-4c29-88a7-5887f1f08f92","Type":"ContainerStarted","Data":"f30a92664b877d861d7ced349eceb4d5516302c0bbbfaf8c2c9aef88e9e6e918"}
Oct 30 20:41:18.493168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.493071    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r" event={"ID":"010a8f71-8fcf-4c29-88a7-5887f1f08f92","Type":"ContainerStarted","Data":"e7648515e252bde2a4ee630c00ba6b8a99ac6100419aa0acaac57b79e59a45a1"}
Oct 30 20:41:18.662077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.662051    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"]
Oct 30 20:41:18.662854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.662831    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1db77f87-7883-4939-980c-6d38a4fa91cd" containerName="revoke-keys"
Oct 30 20:41:18.662854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.662853    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db77f87-7883-4939-980c-6d38a4fa91cd" containerName="revoke-keys"
Oct 30 20:41:18.663060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.663048    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db77f87-7883-4939-980c-6d38a4fa91cd" containerName="revoke-keys"
Oct 30 20:41:18.667300 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.667278    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.675179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.673724    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:18.702770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.702723    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca\") pod \"0553767a-9895-4946-b66c-516203b38291\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") "
Oct 30 20:41:18.703010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.702839    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca\") pod \"0553767a-9895-4946-b66c-516203b38291\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") "
Oct 30 20:41:18.703010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.702960    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4\" (UniqueName: \"kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4\") pod \"0553767a-9895-4946-b66c-516203b38291\" (UID: \"0553767a-9895-4946-b66c-516203b38291\") "
Oct 30 20:41:18.703162 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.703143    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca" pod "0553767a-9895-4946-b66c-516203b38291" (UID: "0553767a-9895-4946-b66c-516203b38291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:18.703329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.703306    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.703395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.703350    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.703395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.703384    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzbf\" (UniqueName: \"kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.703625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.703608    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0553767a-9895-4946-b66c-516203b38291-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.704245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.704209    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"]
Oct 30 20:41:18.710148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.710115    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca" pod "0553767a-9895-4946-b66c-516203b38291" (UID: "0553767a-9895-4946-b66c-516203b38291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:18.710240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.710210    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4" pod "0553767a-9895-4946-b66c-516203b38291" (UID: "0553767a-9895-4946-b66c-516203b38291"). InnerVolumeSpecName "kube-api-access-ms5l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:18.723860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.723816    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r" podStartSLOduration=1.723803217 podStartE2EDuration="1.723803217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:18.720088457 +0000 UTC m=+1259.320890417" watchObservedRunningTime="2026-10-30 20:41:18.723803217 +0000 UTC m=+1259.324605175"
Oct 30 20:41:18.805247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.805212    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.805247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.805247    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.805470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.805269    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzbf\" (UniqueName: \"kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.805470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.805373    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0553767a-9895-4946-b66c-516203b38291-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.805470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.805385    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5l4\" (UniqueName: \"kubernetes.io/projected/0553767a-9895-4946-b66c-516203b38291-kube-api-access-ms5l4\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:18.805915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.805893    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.807647 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.807627    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.861957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.861926    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzbf\" (UniqueName: \"kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf\") pod \"maas-api-revoke-sub-2dccfc9e-mwg2r\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:18.989533 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:18.989499    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:19.056106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.056075    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"]
Oct 30 20:41:19.065792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.065159    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56a95ac-c2nws"]
Oct 30 20:41:19.388138 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.388108    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"]
Oct 30 20:41:19.391774 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:19.391732    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4579070e_3803_49b0_b2dc_4cbd9f726f7e.slice/crio-405f5d5af6ee0b05b887fe502ed0aa1cb699170fbd64a8ca256262286e91b0d1 WatchSource:0}: Error finding container 405f5d5af6ee0b05b887fe502ed0aa1cb699170fbd64a8ca256262286e91b0d1: Status 404 returned error can't find the container with id 405f5d5af6ee0b05b887fe502ed0aa1cb699170fbd64a8ca256262286e91b0d1
Oct 30 20:41:19.502869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.502842    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"
Oct 30 20:41:19.503232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.502869    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd" event={"ID":"0553767a-9895-4946-b66c-516203b38291","Type":"ContainerDied","Data":"92f8337c64e5fbc866c4a1f07ec90b4a87d2779b44c4e11539886634e853d235"}
Oct 30 20:41:19.503232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.502913    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f8337c64e5fbc866c4a1f07ec90b4a87d2779b44c4e11539886634e853d235"
Oct 30 20:41:19.504394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.504371    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r" event={"ID":"4579070e-3803-49b0-b2dc-4cbd9f726f7e","Type":"ContainerStarted","Data":"405f5d5af6ee0b05b887fe502ed0aa1cb699170fbd64a8ca256262286e91b0d1"}
Oct 30 20:41:19.505935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.505911    5967 generic.go:356] "Generic (PLEG): container finished" podID="010a8f71-8fcf-4c29-88a7-5887f1f08f92" containerID="f30a92664b877d861d7ced349eceb4d5516302c0bbbfaf8c2c9aef88e9e6e918" exitCode=0
Oct 30 20:41:19.506037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.505990    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r" event={"ID":"010a8f71-8fcf-4c29-88a7-5887f1f08f92","Type":"ContainerDied","Data":"f30a92664b877d861d7ced349eceb4d5516302c0bbbfaf8c2c9aef88e9e6e918"}
Oct 30 20:41:19.545168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.545137    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cef49d-08cd-4424-9c71-fb6bb8561e26" path="/var/lib/kubelet/pods/c0cef49d-08cd-4424-9c71-fb6bb8561e26/volumes"
Oct 30 20:41:19.668192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.668157    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:19.715887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.715864    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca\") pod \"1937f6d0-aa8f-4061-8e22-9291b1746c50\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") "
Oct 30 20:41:19.716001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.715907    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6\" (UniqueName: \"kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6\") pod \"1937f6d0-aa8f-4061-8e22-9291b1746c50\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") "
Oct 30 20:41:19.716089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.716073    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca\") pod \"1937f6d0-aa8f-4061-8e22-9291b1746c50\" (UID: \"1937f6d0-aa8f-4061-8e22-9291b1746c50\") "
Oct 30 20:41:19.716166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.716144    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca" pod "1937f6d0-aa8f-4061-8e22-9291b1746c50" (UID: "1937f6d0-aa8f-4061-8e22-9291b1746c50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:19.716612 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.716586    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:19.718094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.718069    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca" pod "1937f6d0-aa8f-4061-8e22-9291b1746c50" (UID: "1937f6d0-aa8f-4061-8e22-9291b1746c50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:19.718187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.718092    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6" pod "1937f6d0-aa8f-4061-8e22-9291b1746c50" (UID: "1937f6d0-aa8f-4061-8e22-9291b1746c50"). InnerVolumeSpecName "kube-api-access-8vjm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:19.725615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.723361    5967 scope.go:122] "RemoveContainer" containerID="4fc1776e5e4056220297f7578f01b4aa6f54507adca747a11a2997095320bccb"
Oct 30 20:41:19.739643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.739622    5967 scope.go:122] "RemoveContainer" containerID="863ec317aab14f821738df4ca65ebb83c67def27bfa77c555a91ebdf3d89fef5"
Oct 30 20:41:19.751414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.751398    5967 scope.go:122] "RemoveContainer" containerID="521fda86973eafc41b8642ed6aa5697f422a2613d57d5bae24a963ab657540f6"
Oct 30 20:41:19.818275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.818240    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1937f6d0-aa8f-4061-8e22-9291b1746c50-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:19.818275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:19.818269    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjm6\" (UniqueName: \"kubernetes.io/projected/1937f6d0-aa8f-4061-8e22-9291b1746c50-kube-api-access-8vjm6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:20.563579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:20.563541    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq" event={"ID":"1937f6d0-aa8f-4061-8e22-9291b1746c50","Type":"ContainerDied","Data":"acacccac7bee95e1d245e7f2cb43b3dd130ef26e601bb3bebdc0d19618bd9439"}
Oct 30 20:41:20.564119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:20.563603    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acacccac7bee95e1d245e7f2cb43b3dd130ef26e601bb3bebdc0d19618bd9439"
Oct 30 20:41:20.564119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:20.563703    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"
Oct 30 20:41:20.580469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:20.579704    5967 generic.go:356] "Generic (PLEG): container finished" podID="4579070e-3803-49b0-b2dc-4cbd9f726f7e" containerID="b612cc705855e85a11a3030e640d76284064604c0d197d86f2b9118d9dc0507b" exitCode=0
Oct 30 20:41:20.580469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:20.580376    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r" event={"ID":"4579070e-3803-49b0-b2dc-4cbd9f726f7e","Type":"ContainerDied","Data":"b612cc705855e85a11a3030e640d76284064604c0d197d86f2b9118d9dc0507b"}
Oct 30 20:41:21.596778 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.594938    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:21.701818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.701285    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"
Oct 30 20:41:21.833949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.830728    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca\") pod \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") "
Oct 30 20:41:21.833949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.830854    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn\" (UniqueName: \"kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn\") pod \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") "
Oct 30 20:41:21.833949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.831004    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca\") pod \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\" (UID: \"010a8f71-8fcf-4c29-88a7-5887f1f08f92\") "
Oct 30 20:41:21.893766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.893175    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca" pod "010a8f71-8fcf-4c29-88a7-5887f1f08f92" (UID: "010a8f71-8fcf-4c29-88a7-5887f1f08f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:21.898765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.897340    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca" pod "010a8f71-8fcf-4c29-88a7-5887f1f08f92" (UID: "010a8f71-8fcf-4c29-88a7-5887f1f08f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:21.938767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.937723    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:21.938767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.937782    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010a8f71-8fcf-4c29-88a7-5887f1f08f92-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:21.962817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.947430    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r" event={"ID":"010a8f71-8fcf-4c29-88a7-5887f1f08f92","Type":"ContainerDied","Data":"e7648515e252bde2a4ee630c00ba6b8a99ac6100419aa0acaac57b79e59a45a1"}
Oct 30 20:41:21.962817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:21.947493    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7648515e252bde2a4ee630c00ba6b8a99ac6100419aa0acaac57b79e59a45a1"
Oct 30 20:41:22.018772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.017832    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.022768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.020255    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn" pod "010a8f71-8fcf-4c29-88a7-5887f1f08f92" (UID: "010a8f71-8fcf-4c29-88a7-5887f1f08f92"). InnerVolumeSpecName "kube-api-access-tlthn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:22.074777 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.059383    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlthn\" (UniqueName: \"kubernetes.io/projected/010a8f71-8fcf-4c29-88a7-5887f1f08f92-kube-api-access-tlthn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:22.211767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.211200    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded"
Oct 30 20:41:22.234821 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.220027    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.336811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.334959    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.411093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.406076    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.411093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.406828    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.448043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.445905    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 20:41:22.448043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.446107    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 20:41:22.448043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.447198    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.540771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.535623    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 30 20:41:22.641810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.617854    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.898491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.897826    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r" event={"ID":"4579070e-3803-49b0-b2dc-4cbd9f726f7e","Type":"ContainerDied","Data":"405f5d5af6ee0b05b887fe502ed0aa1cb699170fbd64a8ca256262286e91b0d1"}
Oct 30 20:41:22.898491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.897887    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405f5d5af6ee0b05b887fe502ed0aa1cb699170fbd64a8ca256262286e91b0d1"
Oct 30 20:41:22.905773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.901912    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:23.001765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.998439    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf\" (UniqueName: \"kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf\") pod \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") "
Oct 30 20:41:23.001765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.998546    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca\") pod \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") "
Oct 30 20:41:23.001765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:22.998816    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca\") pod \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\" (UID: \"4579070e-3803-49b0-b2dc-4cbd9f726f7e\") "
Oct 30 20:41:23.009510 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.009466    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca" pod "4579070e-3803-49b0-b2dc-4cbd9f726f7e" (UID: "4579070e-3803-49b0-b2dc-4cbd9f726f7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:23.014975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.014947    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca" pod "4579070e-3803-49b0-b2dc-4cbd9f726f7e" (UID: "4579070e-3803-49b0-b2dc-4cbd9f726f7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:23.020350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.018502    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.020350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.019197    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:41:23.106658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.106613    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.109165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.107983    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:23.109354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.109337    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4579070e-3803-49b0-b2dc-4cbd9f726f7e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:23.142806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.140965    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf" pod "4579070e-3803-49b0-b2dc-4cbd9f726f7e" (UID: "4579070e-3803-49b0-b2dc-4cbd9f726f7e"). InnerVolumeSpecName "kube-api-access-pvzbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:23.225815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.198944    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.257890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.228665    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvzbf\" (UniqueName: \"kubernetes.io/projected/4579070e-3803-49b0-b2dc-4cbd9f726f7e-kube-api-access-pvzbf\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:23.291769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.258106    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.291769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.260852    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:41:23.353878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.321486    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"]
Oct 30 20:41:23.354496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.354331    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.354790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.354765    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:41:23.450771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.447180    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.472785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.454773    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:41:23.474761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.473922    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.500827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.475241    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:41:23.573895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.537839    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:23.573895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.538387    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:41:23.717674 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.715171    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf614417-65jzv"]
Oct 30 20:41:23.738948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.738925    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:41:23.739178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.739166    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:41:23.739342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.739330    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:41:23.797048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.796963    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:41:23.808434 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.808410    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:41:23.809185 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.809164    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:41:23.906141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:23.906113    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"
Oct 30 20:41:24.897325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897293    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"]
Oct 30 20:41:24.897960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897929    5967 cpu_manager.go:414] "containerMap: removing container" podUID="010a8f71-8fcf-4c29-88a7-5887f1f08f92" containerName="revoke-keys"
Oct 30 20:41:24.897960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897942    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010a8f71-8fcf-4c29-88a7-5887f1f08f92" containerName="revoke-keys"
Oct 30 20:41:24.897960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897958    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4579070e-3803-49b0-b2dc-4cbd9f726f7e" containerName="revoke-keys"
Oct 30 20:41:24.897960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897963    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4579070e-3803-49b0-b2dc-4cbd9f726f7e" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897982    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1937f6d0-aa8f-4061-8e22-9291b1746c50" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.897987    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1937f6d0-aa8f-4061-8e22-9291b1746c50" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.898003    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0553767a-9895-4946-b66c-516203b38291" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.898007    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0553767a-9895-4946-b66c-516203b38291" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.898117    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1937f6d0-aa8f-4061-8e22-9291b1746c50" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.898135    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0553767a-9895-4946-b66c-516203b38291" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.898146    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="010a8f71-8fcf-4c29-88a7-5887f1f08f92" containerName="revoke-keys"
Oct 30 20:41:24.898227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.898153    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4579070e-3803-49b0-b2dc-4cbd9f726f7e" containerName="revoke-keys"
Oct 30 20:41:24.902018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.902000    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:24.908836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.908813    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:24.908984 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.908949    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbpr\" (UniqueName: \"kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:24.909134 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.909121    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:24.954971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.954942    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:41:24.955248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.955201    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:41:24.955248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.955211    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:24.955385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.954961    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:41:24.960772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:24.960723    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"]
Oct 30 20:41:25.010484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.010457    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbpr\" (UniqueName: \"kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.010672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.010651    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.010762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.010711    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.011262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.011244    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.013108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.013093    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.067225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.067200    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbpr\" (UniqueName: \"kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr\") pod \"maas-api-revoke-sub-0e99ced4-blwcm\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.217237 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.217214    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:25.543440 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.543374    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf1259a-192e-4271-9bdd-d602cbc7a47e" path="/var/lib/kubelet/pods/baf1259a-192e-4271-9bdd-d602cbc7a47e/volumes"
Oct 30 20:41:25.595349 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:25.595319    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd62776_1b2f_4592_87ee_c8846599692f.slice/crio-b6efa6280911f9a22ebc5adbf136a1628ede9ba5d83186fc2a0fee3842544dad WatchSource:0}: Error finding container b6efa6280911f9a22ebc5adbf136a1628ede9ba5d83186fc2a0fee3842544dad: Status 404 returned error can't find the container with id b6efa6280911f9a22ebc5adbf136a1628ede9ba5d83186fc2a0fee3842544dad
Oct 30 20:41:25.595669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.595653    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"]
Oct 30 20:41:25.923660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.923616    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm" event={"ID":"fdd62776-1b2f-4592-87ee-c8846599692f","Type":"ContainerStarted","Data":"0a003be3606f4eb47b7db2a27fa47e1ae51d0655476cc5ee4686caae2c249cec"}
Oct 30 20:41:25.924115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:25.923675    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm" event={"ID":"fdd62776-1b2f-4592-87ee-c8846599692f","Type":"ContainerStarted","Data":"b6efa6280911f9a22ebc5adbf136a1628ede9ba5d83186fc2a0fee3842544dad"}
Oct 30 20:41:26.022545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.022494    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"]
Oct 30 20:41:26.032303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.032247    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.055884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.055838    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm" podStartSLOduration=2.055825722 podStartE2EDuration="2.055825722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:26.051299295 +0000 UTC m=+1266.652101254" watchObservedRunningTime="2026-10-30 20:41:26.055825722 +0000 UTC m=+1266.656627681"
Oct 30 20:41:26.095995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.095964    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"]
Oct 30 20:41:26.121922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.121900    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"]
Oct 30 20:41:26.125261 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.125237    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.125387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.125366    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.125429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.125405    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fsr2\" (UniqueName: \"kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.136374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.136316    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"]
Oct 30 20:41:26.147392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.147367    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8ed325-p6r88"]
Oct 30 20:41:26.150929 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.150907    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd31e55-bffsm"]
Oct 30 20:41:26.226988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.226958    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.227148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.227002    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fsr2\" (UniqueName: \"kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.227148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.227135    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.227607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.227584    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.229470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.229451    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.285851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.285824    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fsr2\" (UniqueName: \"kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2\") pod \"maas-api-revoke-sub-5f1af342-9kvsm\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.348669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.348638    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:26.731319 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:26.731294    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0348ff2_00b0_4c8d_ae61_0540ac075b2a.slice/crio-72eafd4b999ed8b17fc4a1c7f069f4c1781d06b3a6bf0412944240dbac222fae WatchSource:0}: Error finding container 72eafd4b999ed8b17fc4a1c7f069f4c1781d06b3a6bf0412944240dbac222fae: Status 404 returned error can't find the container with id 72eafd4b999ed8b17fc4a1c7f069f4c1781d06b3a6bf0412944240dbac222fae
Oct 30 20:41:26.731425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.731315    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"]
Oct 30 20:41:26.933240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.933203    5967 generic.go:356] "Generic (PLEG): container finished" podID="fdd62776-1b2f-4592-87ee-c8846599692f" containerID="0a003be3606f4eb47b7db2a27fa47e1ae51d0655476cc5ee4686caae2c249cec" exitCode=0
Oct 30 20:41:26.933657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.933287    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm" event={"ID":"fdd62776-1b2f-4592-87ee-c8846599692f","Type":"ContainerDied","Data":"0a003be3606f4eb47b7db2a27fa47e1ae51d0655476cc5ee4686caae2c249cec"}
Oct 30 20:41:26.934817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:26.934781    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm" event={"ID":"e0348ff2-00b0-4c8d-ae61-0540ac075b2a","Type":"ContainerStarted","Data":"72eafd4b999ed8b17fc4a1c7f069f4c1781d06b3a6bf0412944240dbac222fae"}
Oct 30 20:41:27.174903 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.174865    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"]
Oct 30 20:41:27.179320 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.179297    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.236718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.236658    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"]
Oct 30 20:41:27.239911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.239889    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.240010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.239969    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.240126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.240102    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwx9k\" (UniqueName: \"kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.341106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.341073    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.341255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.341169    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.341255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.341215    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwx9k\" (UniqueName: \"kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.341657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.341631    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.343606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.343576    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.398222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.398199    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwx9k\" (UniqueName: \"kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k\") pod \"maas-api-revoke-sub-6d113c27-k4j5s\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.493326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.493268    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:27.546724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.546690    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb3403f-f568-4a98-855d-90be9bc5ecc0" path="/var/lib/kubelet/pods/0eb3403f-f568-4a98-855d-90be9bc5ecc0/volumes"
Oct 30 20:41:27.547358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.547335    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cab2e9-4ec3-4287-8737-c72a6b75db92" path="/var/lib/kubelet/pods/f1cab2e9-4ec3-4287-8737-c72a6b75db92/volumes"
Oct 30 20:41:27.874691 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.874662    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"]
Oct 30 20:41:27.874943 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:27.874919    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf07786d_5756_4ace_ab26_9081a682d12e.slice/crio-e511cc77d4e7b403a1ca36771c2ba5cd08dd8bba98083e748ddc6baca5bca9c8 WatchSource:0}: Error finding container e511cc77d4e7b403a1ca36771c2ba5cd08dd8bba98083e748ddc6baca5bca9c8: Status 404 returned error can't find the container with id e511cc77d4e7b403a1ca36771c2ba5cd08dd8bba98083e748ddc6baca5bca9c8
Oct 30 20:41:27.944247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.944222    5967 generic.go:356] "Generic (PLEG): container finished" podID="e0348ff2-00b0-4c8d-ae61-0540ac075b2a" containerID="ec22e27721d07b799a36a1472cef80bf58a188fd277ba789863df25ef3736ede" exitCode=0
Oct 30 20:41:27.944623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.944312    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm" event={"ID":"e0348ff2-00b0-4c8d-ae61-0540ac075b2a","Type":"ContainerDied","Data":"ec22e27721d07b799a36a1472cef80bf58a188fd277ba789863df25ef3736ede"}
Oct 30 20:41:27.945821 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:27.945788    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s" event={"ID":"bf07786d-5756-4ace-ab26-9081a682d12e","Type":"ContainerStarted","Data":"e511cc77d4e7b403a1ca36771c2ba5cd08dd8bba98083e748ddc6baca5bca9c8"}
Oct 30 20:41:28.116429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.116406    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:28.152036 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.151978    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr\" (UniqueName: \"kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr\") pod \"fdd62776-1b2f-4592-87ee-c8846599692f\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") "
Oct 30 20:41:28.152143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.152039    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca\") pod \"fdd62776-1b2f-4592-87ee-c8846599692f\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") "
Oct 30 20:41:28.152143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.152100    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca\") pod \"fdd62776-1b2f-4592-87ee-c8846599692f\" (UID: \"fdd62776-1b2f-4592-87ee-c8846599692f\") "
Oct 30 20:41:28.152470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.152446    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca" pod "fdd62776-1b2f-4592-87ee-c8846599692f" (UID: "fdd62776-1b2f-4592-87ee-c8846599692f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:28.152655 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.152641    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:28.154282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.154262    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca" pod "fdd62776-1b2f-4592-87ee-c8846599692f" (UID: "fdd62776-1b2f-4592-87ee-c8846599692f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:28.154376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.154329    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr" pod "fdd62776-1b2f-4592-87ee-c8846599692f" (UID: "fdd62776-1b2f-4592-87ee-c8846599692f"). InnerVolumeSpecName "kube-api-access-mjbpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:28.253851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.253822    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbpr\" (UniqueName: \"kubernetes.io/projected/fdd62776-1b2f-4592-87ee-c8846599692f-kube-api-access-mjbpr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:28.253851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.253848    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd62776-1b2f-4592-87ee-c8846599692f-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:28.954996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.954966    5967 generic.go:356] "Generic (PLEG): container finished" podID="bf07786d-5756-4ace-ab26-9081a682d12e" containerID="f8039721d61ac6687727316022b2b37b51da2b339ee5a3762432bb4ab539bce2" exitCode=0
Oct 30 20:41:28.955430 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.955060    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s" event={"ID":"bf07786d-5756-4ace-ab26-9081a682d12e","Type":"ContainerDied","Data":"f8039721d61ac6687727316022b2b37b51da2b339ee5a3762432bb4ab539bce2"}
Oct 30 20:41:28.956577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.956556    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"
Oct 30 20:41:28.956577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.956559    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm" event={"ID":"fdd62776-1b2f-4592-87ee-c8846599692f","Type":"ContainerDied","Data":"b6efa6280911f9a22ebc5adbf136a1628ede9ba5d83186fc2a0fee3842544dad"}
Oct 30 20:41:28.956735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:28.956613    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6efa6280911f9a22ebc5adbf136a1628ede9ba5d83186fc2a0fee3842544dad"
Oct 30 20:41:29.116617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.116595    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:29.165197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.165173    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2\" (UniqueName: \"kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2\") pod \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") "
Oct 30 20:41:29.165331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.165294    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca\") pod \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") "
Oct 30 20:41:29.165331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.165316    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca\") pod \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\" (UID: \"e0348ff2-00b0-4c8d-ae61-0540ac075b2a\") "
Oct 30 20:41:29.165731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.165695    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca" pod "e0348ff2-00b0-4c8d-ae61-0540ac075b2a" (UID: "e0348ff2-00b0-4c8d-ae61-0540ac075b2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:29.166031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.165988    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:29.167331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.167310    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2" pod "e0348ff2-00b0-4c8d-ae61-0540ac075b2a" (UID: "e0348ff2-00b0-4c8d-ae61-0540ac075b2a"). InnerVolumeSpecName "kube-api-access-8fsr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:29.167551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.167538    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca" pod "e0348ff2-00b0-4c8d-ae61-0540ac075b2a" (UID: "e0348ff2-00b0-4c8d-ae61-0540ac075b2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:29.267696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.267647    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fsr2\" (UniqueName: \"kubernetes.io/projected/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-kube-api-access-8fsr2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:29.267696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.267693    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0348ff2-00b0-4c8d-ae61-0540ac075b2a-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:29.742260 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.742231    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"]
Oct 30 20:41:29.742863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.742845    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e0348ff2-00b0-4c8d-ae61-0540ac075b2a" containerName="revoke-keys"
Oct 30 20:41:29.742863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.742861    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0348ff2-00b0-4c8d-ae61-0540ac075b2a" containerName="revoke-keys"
Oct 30 20:41:29.742950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.742884    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fdd62776-1b2f-4592-87ee-c8846599692f" containerName="revoke-keys"
Oct 30 20:41:29.742950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.742889    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd62776-1b2f-4592-87ee-c8846599692f" containerName="revoke-keys"
Oct 30 20:41:29.743014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.743007    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0348ff2-00b0-4c8d-ae61-0540ac075b2a" containerName="revoke-keys"
Oct 30 20:41:29.743046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.743016    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd62776-1b2f-4592-87ee-c8846599692f" containerName="revoke-keys"
Oct 30 20:41:29.747603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.747588    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.772887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.772864    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.772974 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.772940    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.773058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.773040    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46z45\" (UniqueName: \"kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.803986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.803923    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"]
Oct 30 20:41:29.874547 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.874515    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.874714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.874633    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46z45\" (UniqueName: \"kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.874842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.874819    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.875504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.875483    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.877247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.877226    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.932426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.932407    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46z45\" (UniqueName: \"kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45\") pod \"maas-api-revoke-sub-71882aa2-x74bt\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:29.965971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.965953    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"
Oct 30 20:41:29.966345 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.965992    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm" event={"ID":"e0348ff2-00b0-4c8d-ae61-0540ac075b2a","Type":"ContainerDied","Data":"72eafd4b999ed8b17fc4a1c7f069f4c1781d06b3a6bf0412944240dbac222fae"}
Oct 30 20:41:29.966345 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:29.966037    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72eafd4b999ed8b17fc4a1c7f069f4c1781d06b3a6bf0412944240dbac222fae"
Oct 30 20:41:30.060895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.060799    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:30.136477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.136344    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:30.178473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.178449    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k\" (UniqueName: \"kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k\") pod \"bf07786d-5756-4ace-ab26-9081a682d12e\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") "
Oct 30 20:41:30.178627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.178609    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca\") pod \"bf07786d-5756-4ace-ab26-9081a682d12e\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") "
Oct 30 20:41:30.178802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.178783    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca\") pod \"bf07786d-5756-4ace-ab26-9081a682d12e\" (UID: \"bf07786d-5756-4ace-ab26-9081a682d12e\") "
Oct 30 20:41:30.179156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.179127    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca" pod "bf07786d-5756-4ace-ab26-9081a682d12e" (UID: "bf07786d-5756-4ace-ab26-9081a682d12e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:30.179384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.179301    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:30.180598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.180579    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k" pod "bf07786d-5756-4ace-ab26-9081a682d12e" (UID: "bf07786d-5756-4ace-ab26-9081a682d12e"). InnerVolumeSpecName "kube-api-access-gwx9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:30.180862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.180844    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca" pod "bf07786d-5756-4ace-ab26-9081a682d12e" (UID: "bf07786d-5756-4ace-ab26-9081a682d12e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:30.281084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.281054    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwx9k\" (UniqueName: \"kubernetes.io/projected/bf07786d-5756-4ace-ab26-9081a682d12e-kube-api-access-gwx9k\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:30.281084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.281080    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf07786d-5756-4ace-ab26-9081a682d12e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:30.453828 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:30.453800    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe044cf0_5710_4332_b482_b11b4e03be39.slice/crio-6d08f6c41e893ad3c9a341bc66aebae226f4742c84d63663b5843d3ea55bdd99 WatchSource:0}: Error finding container 6d08f6c41e893ad3c9a341bc66aebae226f4742c84d63663b5843d3ea55bdd99: Status 404 returned error can't find the container with id 6d08f6c41e893ad3c9a341bc66aebae226f4742c84d63663b5843d3ea55bdd99
Oct 30 20:41:30.453971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.453929    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"]
Oct 30 20:41:30.976210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.976184    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"
Oct 30 20:41:30.976584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.976221    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s" event={"ID":"bf07786d-5756-4ace-ab26-9081a682d12e","Type":"ContainerDied","Data":"e511cc77d4e7b403a1ca36771c2ba5cd08dd8bba98083e748ddc6baca5bca9c8"}
Oct 30 20:41:30.976584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.976294    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e511cc77d4e7b403a1ca36771c2ba5cd08dd8bba98083e748ddc6baca5bca9c8"
Oct 30 20:41:30.978000 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.977973    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt" event={"ID":"fe044cf0-5710-4332-b482-b11b4e03be39","Type":"ContainerStarted","Data":"a39ec9185fe324a08f3ef1bf6e59b797e547d68726b4dcf0f9bbdf4adf75d4ce"}
Oct 30 20:41:30.978113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:30.978025    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt" event={"ID":"fe044cf0-5710-4332-b482-b11b4e03be39","Type":"ContainerStarted","Data":"6d08f6c41e893ad3c9a341bc66aebae226f4742c84d63663b5843d3ea55bdd99"}
Oct 30 20:41:31.097760 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:31.097670    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt" podStartSLOduration=2.097656924 podStartE2EDuration="2.097656924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:31.095241974 +0000 UTC m=+1271.696043945" watchObservedRunningTime="2026-10-30 20:41:31.097656924 +0000 UTC m=+1271.698458886"
Oct 30 20:41:31.980875 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:41:31.980840    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:41:31.988199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:31.988175    5967 generic.go:356] "Generic (PLEG): container finished" podID="fe044cf0-5710-4332-b482-b11b4e03be39" containerID="a39ec9185fe324a08f3ef1bf6e59b797e547d68726b4dcf0f9bbdf4adf75d4ce" exitCode=0
Oct 30 20:41:31.988374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:31.988215    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt" event={"ID":"fe044cf0-5710-4332-b482-b11b4e03be39","Type":"ContainerDied","Data":"a39ec9185fe324a08f3ef1bf6e59b797e547d68726b4dcf0f9bbdf4adf75d4ce"}
Oct 30 20:41:33.142881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.142859    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:33.218204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.218178    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca\") pod \"fe044cf0-5710-4332-b482-b11b4e03be39\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") "
Oct 30 20:41:33.218348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.218336    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45\" (UniqueName: \"kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45\") pod \"fe044cf0-5710-4332-b482-b11b4e03be39\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") "
Oct 30 20:41:33.218396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.218371    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca\") pod \"fe044cf0-5710-4332-b482-b11b4e03be39\" (UID: \"fe044cf0-5710-4332-b482-b11b4e03be39\") "
Oct 30 20:41:33.218845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.218714    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca" pod "fe044cf0-5710-4332-b482-b11b4e03be39" (UID: "fe044cf0-5710-4332-b482-b11b4e03be39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:33.220225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.220205    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca" pod "fe044cf0-5710-4332-b482-b11b4e03be39" (UID: "fe044cf0-5710-4332-b482-b11b4e03be39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:33.220425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.220412    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45" pod "fe044cf0-5710-4332-b482-b11b4e03be39" (UID: "fe044cf0-5710-4332-b482-b11b4e03be39"). InnerVolumeSpecName "kube-api-access-46z45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:33.319919 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.319858    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:33.319919 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.319882    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46z45\" (UniqueName: \"kubernetes.io/projected/fe044cf0-5710-4332-b482-b11b4e03be39-kube-api-access-46z45\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:33.319919 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:33.319892    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe044cf0-5710-4332-b482-b11b4e03be39-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:34.010253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:34.010223    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"
Oct 30 20:41:34.010442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:34.010258    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt" event={"ID":"fe044cf0-5710-4332-b482-b11b4e03be39","Type":"ContainerDied","Data":"6d08f6c41e893ad3c9a341bc66aebae226f4742c84d63663b5843d3ea55bdd99"}
Oct 30 20:41:34.010442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:34.010303    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d08f6c41e893ad3c9a341bc66aebae226f4742c84d63663b5843d3ea55bdd99"
Oct 30 20:41:35.944797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.944764    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"]
Oct 30 20:41:35.945398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.945383    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fe044cf0-5710-4332-b482-b11b4e03be39" containerName="revoke-keys"
Oct 30 20:41:35.945441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.945399    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe044cf0-5710-4332-b482-b11b4e03be39" containerName="revoke-keys"
Oct 30 20:41:35.945441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.945415    5967 cpu_manager.go:414] "containerMap: removing container" podUID="bf07786d-5756-4ace-ab26-9081a682d12e" containerName="revoke-keys"
Oct 30 20:41:35.945441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.945420    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf07786d-5756-4ace-ab26-9081a682d12e" containerName="revoke-keys"
Oct 30 20:41:35.945550 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.945541    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe044cf0-5710-4332-b482-b11b4e03be39" containerName="revoke-keys"
Oct 30 20:41:35.945589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.945553    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf07786d-5756-4ace-ab26-9081a682d12e" containerName="revoke-keys"
Oct 30 20:41:35.949266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:35.949249    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.004708 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.004681    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:36.053990 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.053961    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.054145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.054017    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.054145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.054072    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jv9m\" (UniqueName: \"kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.054414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.054395    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:41:36.054497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.054479    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:41:36.058639 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.058611    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"]
Oct 30 20:41:36.106476 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.106445    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:41:36.155462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.155436    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.155591 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.155474    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jv9m\" (UniqueName: \"kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.155673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.155662    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.156042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.156024    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.158062 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.158039    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.220735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.220717    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jv9m\" (UniqueName: \"kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m\") pod \"maas-api-revoke-sub-d6080963-vwlmn\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.262695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.262677    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:36.644720 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:36.644688    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d634662_f0c0_47d4_b9d1_c001173afcc0.slice/crio-4fe5f806a85212e8323c4e244ebaae9faa18a762c2f5058e71b3b05f4693cf91 WatchSource:0}: Error finding container 4fe5f806a85212e8323c4e244ebaae9faa18a762c2f5058e71b3b05f4693cf91: Status 404 returned error can't find the container with id 4fe5f806a85212e8323c4e244ebaae9faa18a762c2f5058e71b3b05f4693cf91
Oct 30 20:41:36.644885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:36.644848    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"]
Oct 30 20:41:37.038464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:37.038434    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn" event={"ID":"2d634662-f0c0-47d4-b9d1-c001173afcc0","Type":"ContainerStarted","Data":"78ff5f17cff6605bbabaf0a6d972862da5aa7fd87e4b31f131acd54b5f594391"}
Oct 30 20:41:37.038869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:37.038492    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn" event={"ID":"2d634662-f0c0-47d4-b9d1-c001173afcc0","Type":"ContainerStarted","Data":"4fe5f806a85212e8323c4e244ebaae9faa18a762c2f5058e71b3b05f4693cf91"}
Oct 30 20:41:37.048075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:37.048052    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"]
Oct 30 20:41:37.085216 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:37.085196    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c80372-s9k78"]
Oct 30 20:41:37.160496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:37.160447    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn" podStartSLOduration=2.1604343520000002 podStartE2EDuration="2.160434352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:37.156358618 +0000 UTC m=+1277.757160578" watchObservedRunningTime="2026-10-30 20:41:37.160434352 +0000 UTC m=+1277.761236311"
Oct 30 20:41:37.544270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:37.544241    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7156d30b-5884-4ea5-9b0d-4e37a6b6db02" path="/var/lib/kubelet/pods/7156d30b-5884-4ea5-9b0d-4e37a6b6db02/volumes"
Oct 30 20:41:38.048596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:38.048564    5967 generic.go:356] "Generic (PLEG): container finished" podID="2d634662-f0c0-47d4-b9d1-c001173afcc0" containerID="78ff5f17cff6605bbabaf0a6d972862da5aa7fd87e4b31f131acd54b5f594391" exitCode=0
Oct 30 20:41:38.048994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:38.048644    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn" event={"ID":"2d634662-f0c0-47d4-b9d1-c001173afcc0","Type":"ContainerDied","Data":"78ff5f17cff6605bbabaf0a6d972862da5aa7fd87e4b31f131acd54b5f594391"}
Oct 30 20:41:39.225435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.225408    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:39.294518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.294486    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m\" (UniqueName: \"kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m\") pod \"2d634662-f0c0-47d4-b9d1-c001173afcc0\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") "
Oct 30 20:41:39.294659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.294602    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca\") pod \"2d634662-f0c0-47d4-b9d1-c001173afcc0\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") "
Oct 30 20:41:39.294734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.294705    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca\") pod \"2d634662-f0c0-47d4-b9d1-c001173afcc0\" (UID: \"2d634662-f0c0-47d4-b9d1-c001173afcc0\") "
Oct 30 20:41:39.294927 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.294905    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca" pod "2d634662-f0c0-47d4-b9d1-c001173afcc0" (UID: "2d634662-f0c0-47d4-b9d1-c001173afcc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:39.295215 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.295197    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:39.296688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.296665    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca" pod "2d634662-f0c0-47d4-b9d1-c001173afcc0" (UID: "2d634662-f0c0-47d4-b9d1-c001173afcc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:39.296769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.296690    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m" pod "2d634662-f0c0-47d4-b9d1-c001173afcc0" (UID: "2d634662-f0c0-47d4-b9d1-c001173afcc0"). InnerVolumeSpecName "kube-api-access-6jv9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:39.397278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.397201    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jv9m\" (UniqueName: \"kubernetes.io/projected/2d634662-f0c0-47d4-b9d1-c001173afcc0-kube-api-access-6jv9m\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:39.397278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:39.397232    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d634662-f0c0-47d4-b9d1-c001173afcc0-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:40.015897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.015869    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"]
Oct 30 20:41:40.016507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.016489    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2d634662-f0c0-47d4-b9d1-c001173afcc0" containerName="revoke-keys"
Oct 30 20:41:40.016507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.016504    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d634662-f0c0-47d4-b9d1-c001173afcc0" containerName="revoke-keys"
Oct 30 20:41:40.016680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.016629    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d634662-f0c0-47d4-b9d1-c001173afcc0" containerName="revoke-keys"
Oct 30 20:41:40.020322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.020306    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.161281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.086588    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8660b337-openshift-default-dockercfg-gvj8p\""
Oct 30 20:41:40.161526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.106629    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.161681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.161666    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.161816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.161803    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.161916 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.161902    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.162096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.162075    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.162278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.162261    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfx7x\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.162549 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.162532    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.162657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.162644    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.162806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.162789    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.163086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.139270    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn" event={"ID":"2d634662-f0c0-47d4-b9d1-c001173afcc0","Type":"ContainerDied","Data":"4fe5f806a85212e8323c4e244ebaae9faa18a762c2f5058e71b3b05f4693cf91"}
Oct 30 20:41:40.163230 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.163213    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe5f806a85212e8323c4e244ebaae9faa18a762c2f5058e71b3b05f4693cf91"
Oct 30 20:41:40.163327 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.163315    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"]
Oct 30 20:41:40.163451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.139473    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"
Oct 30 20:41:40.264109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264076    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfx7x\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264187    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264208    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264236    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264297    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264319    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264335    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264351    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.264530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264412    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.265004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.264609    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.265085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.265052    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.265310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.265290    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.265488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.265458    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.265598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.265490    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.267000 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.266954    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.267181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.267162    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.326666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.326628    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfx7x\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.334926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.334896    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"]
Oct 30 20:41:40.335140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.335116    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token\") pod \"e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") " pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.337540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.337511    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:40.345456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.345432    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.365947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.365914    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366099    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366135    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366163    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366317    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366371    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x7v\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366478    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366583    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.366657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.366642    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.420259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.420176    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"]
Oct 30 20:41:40.468791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.468761    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.469009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.468986    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.469158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.469139    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.469380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.469311    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.469546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.469529    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x7v\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.469711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.469693    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.469938 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.469920    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.470066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.470052    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.470196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.470181    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.475588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.470968    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.477064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.476190    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.477064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.476911    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.477064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.477031    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.477351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.477296    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.477689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.477641    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.479774 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.479731    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.529524 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.529440    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x7v\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.529977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.529950    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.707058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.707019    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:40.823289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.823261    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"]
Oct 30 20:41:40.825789 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:40.825758    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3903830d_6c6a_4f62_9f1a_9ed15aea1d41.slice/crio-c87773b5dcc07f512a7aa5606842b6cf19ad347e7de503197763d7281fcf1a8d WatchSource:0}: Error finding container c87773b5dcc07f512a7aa5606842b6cf19ad347e7de503197763d7281fcf1a8d: Status 404 returned error can't find the container with id c87773b5dcc07f512a7aa5606842b6cf19ad347e7de503197763d7281fcf1a8d
Oct 30 20:41:40.829150 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.829117    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:40.829240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.829195    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:40.829310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:40.829246    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.116067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.116025    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"]
Oct 30 20:41:41.120928 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:41.120894    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaf2514_421d_4d33_9e82_1559bf65c464.slice/crio-63932b02f9806d2f21c5b772a38a6b9274d2868f57a7a250a54c704eba56a83d WatchSource:0}: Error finding container 63932b02f9806d2f21c5b772a38a6b9274d2868f57a7a250a54c704eba56a83d: Status 404 returned error can't find the container with id 63932b02f9806d2f21c5b772a38a6b9274d2868f57a7a250a54c704eba56a83d
Oct 30 20:41:41.124123 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.124091    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.124210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.124155    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.124210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.124184    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:41.152807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.152779    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" event={"ID":"3903830d-6c6a-4f62-9f1a-9ed15aea1d41","Type":"ContainerStarted","Data":"8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"}
Oct 30 20:41:41.152889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.152829    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" event={"ID":"3903830d-6c6a-4f62-9f1a-9ed15aea1d41","Type":"ContainerStarted","Data":"c87773b5dcc07f512a7aa5606842b6cf19ad347e7de503197763d7281fcf1a8d"}
Oct 30 20:41:41.154403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.154380    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" event={"ID":"ebaf2514-421d-4d33-9e82-1559bf65c464","Type":"ContainerStarted","Data":"63932b02f9806d2f21c5b772a38a6b9274d2868f57a7a250a54c704eba56a83d"}
Oct 30 20:41:41.305412 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.305324    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podStartSLOduration=2.305304262 podStartE2EDuration="2.305304262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:41.296944515 +0000 UTC m=+1281.897746492" watchObservedRunningTime="2026-10-30 20:41:41.305304262 +0000 UTC m=+1281.906106223"
Oct 30 20:41:41.337827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.337792    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:41.339352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.339322    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:41:41.339520 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:41.339368    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:41:42.091827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.087407    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"]
Oct 30 20:41:42.097366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.097314    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918014c3-txqpx"]
Oct 30 20:41:42.165481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.165445    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" event={"ID":"ebaf2514-421d-4d33-9e82-1559bf65c464","Type":"ContainerStarted","Data":"36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a"}
Oct 30 20:41:42.302640 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.302587    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podStartSLOduration=2.302573636 podStartE2EDuration="2.302573636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:42.300899852 +0000 UTC m=+1282.901701814" watchObservedRunningTime="2026-10-30 20:41:42.302573636 +0000 UTC m=+1282.903375595"
Oct 30 20:41:42.342284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.342214    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:42.707355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.707285    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:42.709002 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.708979    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:41:42.709108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:42.709025    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 20:41:43.211765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:43.211603    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:43.215761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:43.212615    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:41:43.546245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:43.546199    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7517642-8816-485b-96e1-fad2f8198748" path="/var/lib/kubelet/pods/e7517642-8816-485b-96e1-fad2f8198748/volumes"
Oct 30 20:41:43.708004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:43.707972    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:41:43.708184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:43.708030    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 20:41:44.707377 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:44.707347    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:41:44.707839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:44.707397    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 20:41:45.533255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.533225    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"]
Oct 30 20:41:45.538860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.538837    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.598764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.597988    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-23cde229-openshift-default-dockercfg-h58pv\""
Oct 30 20:41:45.645601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.645565    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.645818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.645645    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.645818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.645676    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.646096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.646062    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.646347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.646301    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.647622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.647600    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.647877 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.647860    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.648065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.648048    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhhq\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.648229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.648214    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.655133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.655101    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"]
Oct 30 20:41:45.708293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.708260    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:41:45.708927 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.708893    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 20:41:45.749309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.749281    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.749542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.749528    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhhq\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.749803 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.749783    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.749933 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.749923    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750073    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750146    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750289    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750367    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750313    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750475    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750646    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750717 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750705    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.750988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.750970    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.751504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.751479    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.753483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.753463    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.753820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.753799    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.807463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.807377    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhhq\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.814356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.814327    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token\") pod \"e2e-ait-23cde229-openshift-default-659c875689-4fjrs\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") " pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:45.857117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:45.857089    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:46.339441 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:46.339411    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef7d8ed_4ba6_4117_a0f7_76165a09e72b.slice/crio-96f0d83d6286bb09a6fbef5fccd61ca0194250de17bf9932988c6b1ff722370f WatchSource:0}: Error finding container 96f0d83d6286bb09a6fbef5fccd61ca0194250de17bf9932988c6b1ff722370f: Status 404 returned error can't find the container with id 96f0d83d6286bb09a6fbef5fccd61ca0194250de17bf9932988c6b1ff722370f
Oct 30 20:41:46.339635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.339614    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"]
Oct 30 20:41:46.341922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.341892    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:46.342005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.341954    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:46.342005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.341985    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:46.707577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.707485    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:41:46.707577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.707549    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 20:41:46.803298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.803262    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"]
Oct 30 20:41:46.809601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.809575    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.861167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.861136    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:41:46.861346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.861282    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:46.861417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.861339    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:41:46.861494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.861466    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:41:46.863383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.863359    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"]
Oct 30 20:41:46.865262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.865240    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.865367 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.865351    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrp9\" (UniqueName: \"kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.865544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.865521    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.966974 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.966950    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrp9\" (UniqueName: \"kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.967126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.967091    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.967165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.967144    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.967779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.967733    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:46.984770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:46.983127    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:47.025488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.025453    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrp9\" (UniqueName: \"kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9\") pod \"maas-api-revoke-sub-92c55f38-6bvp5\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:47.127175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.127147    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:47.255712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.255673    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" event={"ID":"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b","Type":"ContainerStarted","Data":"8e08bdb41706159fbf1e29d725bc613db7c4f42c90b71306c72500f2e99e05e3"}
Oct 30 20:41:47.255951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.255772    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" event={"ID":"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b","Type":"ContainerStarted","Data":"96f0d83d6286bb09a6fbef5fccd61ca0194250de17bf9932988c6b1ff722370f"}
Oct 30 20:41:47.384834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.384772    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podStartSLOduration=2.384734443 podStartE2EDuration="2.384734443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:47.383050302 +0000 UTC m=+1287.983852263" watchObservedRunningTime="2026-10-30 20:41:47.384734443 +0000 UTC m=+1287.985536404"
Oct 30 20:41:47.527641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.527615    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"]
Oct 30 20:41:47.527808 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:47.527623    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3384efb9_738f_4d80_950b_dbdf4a15010d.slice/crio-a4f2f2d35b3aa551ba6e799cc6b5a81e897d2ce17d3245cd04a77f1de2c041aa WatchSource:0}: Error finding container a4f2f2d35b3aa551ba6e799cc6b5a81e897d2ce17d3245cd04a77f1de2c041aa: Status 404 returned error can't find the container with id a4f2f2d35b3aa551ba6e799cc6b5a81e897d2ce17d3245cd04a77f1de2c041aa
Oct 30 20:41:47.708619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.708150    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 20:41:47.708619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.708229    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 20:41:47.857623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.857532    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:47.869645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:47.869609    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:48.268332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.268297    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5" event={"ID":"3384efb9-738f-4d80-950b-dbdf4a15010d","Type":"ContainerStarted","Data":"74308487e81ee40e3c29d55508eecc130804ce664af24822e2f571cc560f14b0"}
Oct 30 20:41:48.268537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.268354    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5" event={"ID":"3384efb9-738f-4d80-950b-dbdf4a15010d","Type":"ContainerStarted","Data":"a4f2f2d35b3aa551ba6e799cc6b5a81e897d2ce17d3245cd04a77f1de2c041aa"}
Oct 30 20:41:48.268537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.268383    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:48.269627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.269601    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:41:48.385857 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.385791    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5" podStartSLOduration=2.38577244 podStartE2EDuration="2.38577244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:48.383000922 +0000 UTC m=+1288.983802884" watchObservedRunningTime="2026-10-30 20:41:48.38577244 +0000 UTC m=+1288.986574400"
Oct 30 20:41:48.712756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.712709    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:48.713129 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.713102    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:48.713811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:48.713796    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:41:49.281244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:49.281213    5967 generic.go:356] "Generic (PLEG): container finished" podID="3384efb9-738f-4d80-950b-dbdf4a15010d" containerID="74308487e81ee40e3c29d55508eecc130804ce664af24822e2f571cc560f14b0" exitCode=0
Oct 30 20:41:49.281696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:49.281313    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5" event={"ID":"3384efb9-738f-4d80-950b-dbdf4a15010d","Type":"ContainerDied","Data":"74308487e81ee40e3c29d55508eecc130804ce664af24822e2f571cc560f14b0"}
Oct 30 20:41:50.476307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.476286    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:50.534031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.533999    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca\") pod \"3384efb9-738f-4d80-950b-dbdf4a15010d\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") "
Oct 30 20:41:50.534198 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.534071    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9\" (UniqueName: \"kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9\") pod \"3384efb9-738f-4d80-950b-dbdf4a15010d\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") "
Oct 30 20:41:50.534320 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.534307    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca\") pod \"3384efb9-738f-4d80-950b-dbdf4a15010d\" (UID: \"3384efb9-738f-4d80-950b-dbdf4a15010d\") "
Oct 30 20:41:50.534463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.534441    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca" pod "3384efb9-738f-4d80-950b-dbdf4a15010d" (UID: "3384efb9-738f-4d80-950b-dbdf4a15010d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:50.534769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.534734    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:50.536459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.536436    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9" pod "3384efb9-738f-4d80-950b-dbdf4a15010d" (UID: "3384efb9-738f-4d80-950b-dbdf4a15010d"). InnerVolumeSpecName "kube-api-access-xvrp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:50.536601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.536527    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca" pod "3384efb9-738f-4d80-950b-dbdf4a15010d" (UID: "3384efb9-738f-4d80-950b-dbdf4a15010d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:50.636325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.636295    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrp9\" (UniqueName: \"kubernetes.io/projected/3384efb9-738f-4d80-950b-dbdf4a15010d-kube-api-access-xvrp9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:50.636325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:50.636327    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3384efb9-738f-4d80-950b-dbdf4a15010d-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:51.299145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:51.299115    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"
Oct 30 20:41:51.299318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:51.299142    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5" event={"ID":"3384efb9-738f-4d80-950b-dbdf4a15010d","Type":"ContainerDied","Data":"a4f2f2d35b3aa551ba6e799cc6b5a81e897d2ce17d3245cd04a77f1de2c041aa"}
Oct 30 20:41:51.299318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:51.299186    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4f2f2d35b3aa551ba6e799cc6b5a81e897d2ce17d3245cd04a77f1de2c041aa"
Oct 30 20:41:52.508143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.508104    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"]
Oct 30 20:41:52.508884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.508865    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3384efb9-738f-4d80-950b-dbdf4a15010d" containerName="revoke-keys"
Oct 30 20:41:52.508884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.508883    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3384efb9-738f-4d80-950b-dbdf4a15010d" containerName="revoke-keys"
Oct 30 20:41:52.509087 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.509073    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3384efb9-738f-4d80-950b-dbdf4a15010d" containerName="revoke-keys"
Oct 30 20:41:52.513011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.512994    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.556177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.556156    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"]
Oct 30 20:41:52.569666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.569642    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"]
Oct 30 20:41:52.569824 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.569798    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.573993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.573975    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.625653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.625631    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8660b337\""
Oct 30 20:41:52.625905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.625709    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8660b337-dockercfg-9b64s\""
Oct 30 20:41:52.643848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.643818    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"]
Oct 30 20:41:52.692783 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.684611    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.700867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.700834    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.701172 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.701152    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64889\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.704855    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.704919    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.704959    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705009    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8l2j\" (UniqueName: \"kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j\") pod \"payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705043    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705072    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705134    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705176    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705200    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705301    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705328    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705481    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705521    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705588    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6wq\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705673    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705770    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.706836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.705813    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.784344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.784191    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"]
Oct 30 20:41:52.790822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.790776    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"]
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807403    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807463    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807509    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8l2j\" (UniqueName: \"kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j\") pod \"payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807540    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807564    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807603    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807641    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807667    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807715    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807775    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807879    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807917    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.807986    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6wq\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.808494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.808042    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.809377    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.809439    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.809519    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.809543    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.809601    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64889\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.809627    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.810131    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.810438    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.810665    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.811494    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.813436    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.815693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.815141    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.816325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.811493    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.818570 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.816723    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.818570 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.816993    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.818570 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.817772    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.818570 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.818474    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.818838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.818600    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.820712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.819118    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.820842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.820758    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.832368 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.832324    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.867054 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.867029    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8l2j\" (UniqueName: \"kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j\") pod \"payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:52.868031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.868004    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.868118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.868069    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.870629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.870612    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6wq\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:52.873218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.873199    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64889\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.885627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.885605    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:52.892154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:52.892132    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:53.052994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.052908    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"]
Oct 30 20:41:53.061116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.061084    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9539494-5tfrq"]
Oct 30 20:41:53.127269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.127236    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:53.301484 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:53.301443    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d3ffe8_d8ed_4425_bf2d_824bb41af110.slice/crio-4b5ed928264847457f6bcaf531242a1f5f1c5c64c79bb3879b8121d59d4ebd41 WatchSource:0}: Error finding container 4b5ed928264847457f6bcaf531242a1f5f1c5c64c79bb3879b8121d59d4ebd41: Status 404 returned error can't find the container with id 4b5ed928264847457f6bcaf531242a1f5f1c5c64c79bb3879b8121d59d4ebd41
Oct 30 20:41:53.302208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.302185    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"]
Oct 30 20:41:53.304223 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.304190    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.304274 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.304258    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.304314 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.304290    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.334993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.334961    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" event={"ID":"a4d3ffe8-d8ed-4425-bf2d-824bb41af110","Type":"ContainerStarted","Data":"4b5ed928264847457f6bcaf531242a1f5f1c5c64c79bb3879b8121d59d4ebd41"}
Oct 30 20:41:53.339342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.339321    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"]
Oct 30 20:41:53.344031 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:53.344006    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29a78b7_1351_4fe0_bcca_29ed0a717347.slice/crio-51ef966a8853ab67b6b3fad65f6a364b88395277c17ac3384571210c76241e76 WatchSource:0}: Error finding container 51ef966a8853ab67b6b3fad65f6a364b88395277c17ac3384571210c76241e76: Status 404 returned error can't find the container with id 51ef966a8853ab67b6b3fad65f6a364b88395277c17ac3384571210c76241e76
Oct 30 20:41:53.531508 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.531480    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"]
Oct 30 20:41:53.532273 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:41:53.532244    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6bd0c7_1900_429e_aa72_55bacb5f12b2.slice/crio-85685ce4bf3c26b4a8f5bb6065d5ccadbe147fb13d9d21cdbf6e855f4291967c WatchSource:0}: Error finding container 85685ce4bf3c26b4a8f5bb6065d5ccadbe147fb13d9d21cdbf6e855f4291967c: Status 404 returned error can't find the container with id 85685ce4bf3c26b4a8f5bb6065d5ccadbe147fb13d9d21cdbf6e855f4291967c
Oct 30 20:41:53.534825 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.534793    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.534893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.534857    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.534893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.534886    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:53.544087 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:53.544059    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076824e6-bf71-47d3-bdfa-6faf285dd3b9" path="/var/lib/kubelet/pods/076824e6-bf71-47d3-bdfa-6faf285dd3b9/volumes"
Oct 30 20:41:54.350387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.350345    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" event={"ID":"4c6bd0c7-1900-429e-aa72-55bacb5f12b2","Type":"ContainerStarted","Data":"21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7"}
Oct 30 20:41:54.350582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.350413    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" event={"ID":"4c6bd0c7-1900-429e-aa72-55bacb5f12b2","Type":"ContainerStarted","Data":"85685ce4bf3c26b4a8f5bb6065d5ccadbe147fb13d9d21cdbf6e855f4291967c"}
Oct 30 20:41:54.354623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.353600    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" event={"ID":"a4d3ffe8-d8ed-4425-bf2d-824bb41af110","Type":"ContainerStarted","Data":"20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f"}
Oct 30 20:41:54.362761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.362718    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" event={"ID":"f29a78b7-1351-4fe0-bcca-29ed0a717347","Type":"ContainerStarted","Data":"31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7"}
Oct 30 20:41:54.363078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.362786    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" event={"ID":"f29a78b7-1351-4fe0-bcca-29ed0a717347","Type":"ContainerStarted","Data":"51ef966a8853ab67b6b3fad65f6a364b88395277c17ac3384571210c76241e76"}
Oct 30 20:41:54.363979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.363960    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:41:54.640905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.640789    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podStartSLOduration=2.640729543 podStartE2EDuration="2.640729543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.64056191 +0000 UTC m=+1295.241363869" watchObservedRunningTime="2026-10-30 20:41:54.640729543 +0000 UTC m=+1295.241531504"
Oct 30 20:41:54.642007 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.641965    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podStartSLOduration=2.6419545700000002 podStartE2EDuration="2.64195457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.49470277 +0000 UTC m=+1295.095504731" watchObservedRunningTime="2026-10-30 20:41:54.64195457 +0000 UTC m=+1295.242756532"
Oct 30 20:41:54.795765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.791004    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" podStartSLOduration=2.790983696 podStartE2EDuration="2.790983696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.788493983 +0000 UTC m=+1295.389295945" watchObservedRunningTime="2026-10-30 20:41:54.790983696 +0000 UTC m=+1295.391785657"
Oct 30 20:41:54.906823 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.903034    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:41:54.910890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.903081    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:41:54.911158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:54.911127    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:41:55.144780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:55.141355    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:41:55.144780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:55.141511    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:41:55.144780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:55.141557    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:41:55.886199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:55.886167    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:41:55.886890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:55.886861    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:41:56.131922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.131885    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:41:56.132339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.132292    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:41:56.676871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.676829    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:41:56.679102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.679067    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:41:56.694977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.694913    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.719344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.717835    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.810638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.809859    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.818224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.816840    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.818224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.817194    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 30 20:41:56.890654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.890596    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:41:56.891420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.891326    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:41:56.939335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:56.939229    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded"
Oct 30 20:41:57.109289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.109243    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.129664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.129626    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:41:57.130015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.129980    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:41:57.387766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.387381    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.436297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.436250    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.677932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.677836    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.678809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.678787    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:41:57.684850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.684820    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.685439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.685417    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:41:57.699860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.699811    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.700373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.700350    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:41:57.723789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.720821    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.723789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.721837    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:41:57.813870 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.810138    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.847765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.839756    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:41:57.847765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.840697    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.847765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.841050    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:41:57.848115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.848086    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.871766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.848512    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:41:57.962361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:57.942386    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:41:58.025885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.025844    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:41:58.198111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.198057    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:41:58.200218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.198237    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:41:58.200218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.200159    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:41:58.227451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.226416    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:41:58.311806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.311330    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:41:58.394650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.390236    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.394650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.394235    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:41:58.445123 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.445080    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.446097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.446074    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:41:58.701831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.687924    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.732839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.702565    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.733202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.733154    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.852613 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.852572    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.867863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.855791    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 30 20:41:58.945928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.945887    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.005839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.988441    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.005839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:58.988499    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:41:59.075819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.058553    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"]
Oct 30 20:41:59.235994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.235960    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.243383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.243135    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:41:59.243604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.241227    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.259853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.237075    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:41:59.284765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.282714    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.367692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.352274    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-23cde229\""
Oct 30 20:41:59.480051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.480011    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.495776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.487975    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"]
Oct 30 20:41:59.709867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.682616    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447zg\" (UniqueName: \"kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg\") pod \"payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:41:59.741807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.710408    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.844158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.814064    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.859804 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:41:59.859466    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:00.057767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.015298    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.058699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.047301    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.085241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.085192    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.156050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.155013    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:42:00.156050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.155078    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:42:00.278766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.278261    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.316947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.307436    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447zg\" (UniqueName: \"kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg\") pod \"payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:00.316947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.307642    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:00.316947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.308807    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:00.364455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.355478    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:42:00.364455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.355533    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:42:00.364455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.356483    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.466165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.386361    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.466165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.397940    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.466165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.392369    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.492764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.491145    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.588017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.509539    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:00.718833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.655922    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-23cde229-dockercfg-d9hkn\""
Oct 30 20:42:00.718833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.697722    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume\") pod \"payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") " pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:00.718833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.697816    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qrx\" (UniqueName: \"kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx\") pod \"payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") " pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:00.825159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.824450    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume\") pod \"payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") " pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:00.825159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.824522    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qrx\" (UniqueName: \"kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx\") pod \"payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") " pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:00.840200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.839435    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.847780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.841794    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume\") pod \"payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") " pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:00.848074 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.842203    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.035884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:00.998926    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447zg\" (UniqueName: \"kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg\") pod \"payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:01.065419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.065367    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 30 20:42:01.098334 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.087160    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:01.102196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.066156    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 30 20:42:01.102196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.099181    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.288701 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.287830    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.356977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.356822    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.368780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.368654    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"]
Oct 30 20:42:01.374781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.374402    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.374781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.374468    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:42:01.464765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.461302    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.464765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.461359    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:42:01.480780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.471815    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.491604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.491568    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 20:42:01.524435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.524399    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qrx\" (UniqueName: \"kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx\") pod \"payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") " pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:01.700549 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.700518    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:01.890797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.890132    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.890797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.890191    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 20:42:01.933187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933105    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"]
Oct 30 20:42:01.933348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933198    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"]
Oct 30 20:42:01.933348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933310    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 20:42:01.933466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933439    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 20:42:01.933519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933496    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 20:42:01.933576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933565    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 20:42:01.933625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933612    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 20:42:01.933676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933664    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:42:01.933785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.933730    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8"
Oct 30 20:42:01.937694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.936121    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:42:01.937694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.936215    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 20:42:01.937694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:01.937486    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" containerID="cri-o://739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7" gracePeriod=30
Oct 30 20:42:02.022426 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:42:02.022376    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:42:02.127626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.127593    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:42:02.128141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.127637    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 20:42:02.265845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.265816    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"]
Oct 30 20:42:02.384588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.384246    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"]
Oct 30 20:42:02.917396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.917307    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" event={"ID":"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd","Type":"ContainerStarted","Data":"6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba"}
Oct 30 20:42:02.917645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.917622    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" event={"ID":"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd","Type":"ContainerStarted","Data":"28051f80a267877501e12e4cbda45b4af7d8f0ee85c10730ea581af8516c5b17"}
Oct 30 20:42:02.918606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.918584    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:02.923301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.923275    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr" event={"ID":"ddd13fe6-31fc-4690-92cc-bea3ad21430a","Type":"ContainerStarted","Data":"82c3b345f8492fec702395f8b345791a1f6db69b933aae0fd9a99fb18a35e404"}
Oct 30 20:42:02.923466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.923449    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr" event={"ID":"ddd13fe6-31fc-4690-92cc-bea3ad21430a","Type":"ContainerStarted","Data":"1851e8701eadfc421b3d528cde052be116840f7fc8bd9eb48c89e91a38b741bc"}
Oct 30 20:42:02.923635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:02.923619    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:03.407908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.407454    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:42:03.408621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.408597    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:42:03.409727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.409703    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:42:03.415138 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.415115    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:42:03.415601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.415581    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:42:03.416544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.416524    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:42:03.760376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:03.760323    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" podStartSLOduration=5.760311192 podStartE2EDuration="5.760311192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:03.757872161 +0000 UTC m=+1304.358674124" watchObservedRunningTime="2026-10-30 20:42:03.760311192 +0000 UTC m=+1304.361113172"
Oct 30 20:42:04.016536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:04.009268    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr" podStartSLOduration=6.009247875 podStartE2EDuration="6.009247875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:04.008516143 +0000 UTC m=+1304.609318103" watchObservedRunningTime="2026-10-30 20:42:04.009247875 +0000 UTC m=+1304.610049836"
Oct 30 20:42:04.718871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:04.718837    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:42:04.719344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:04.719313    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:42:05.437834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:05.437804    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:42:07.489754 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.489697    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"]
Oct 30 20:42:07.495259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.495231    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.556636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.556593    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.556832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.556668    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.556832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.556721    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.556941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.556856    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.556994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.556955    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.557084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.557051    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.557222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.557188    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrvl6\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.568092 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.568061    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"]
Oct 30 20:42:07.596057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.596028    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:42:07.596270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.596028    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:42:07.596390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.596029    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:42:07.596390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.596138    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jtvwq\""
Oct 30 20:42:07.650806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.650780    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:42:07.659675 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.659643    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.659941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.659913    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.660058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.659970    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.660122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.660105    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.660270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.660251    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.660384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.660361    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.660448 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.660364    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.660512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.660496    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrvl6\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.663444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.662895    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.664474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.664449    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.667333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.667309    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.667612 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.667587    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.716792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.716758    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.717392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.717375    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrvl6\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pzcsw\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.761805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.761778    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.761989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.761877    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.761989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.761927    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.761997    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762059    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:07.762187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762152    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762215    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762296 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762256    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762268    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:07.762352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762301    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762320    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:07.762506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762438    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:07.762506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762453    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert\") pod \"74f553a9-eae7-44da-abf0-5f6181ea9656\" (UID: \"74f553a9-eae7-44da-abf0-5f6181ea9656\") "
Oct 30 20:42:07.762780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.762758    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:07.763450 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.763415    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f553a9-eae7-44da-abf0-5f6181ea9656-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.763608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.763454    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.763608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.763470    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.763608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.763484    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.763608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.763499    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.764796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.764775    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "kube-api-access-8hvbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:07.764942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.764920    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:07.765044 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.764948    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:42:07.765044 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.764962    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token" pod "74f553a9-eae7-44da-abf0-5f6181ea9656" (UID: "74f553a9-eae7-44da-abf0-5f6181ea9656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:07.810477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.810447    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:07.865685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.865651    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.865685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.865684    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvbs\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-kube-api-access-8hvbs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.865685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.865694    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.865937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.865703    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f553a9-eae7-44da-abf0-5f6181ea9656-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:07.981988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.981960    5967 generic.go:356] "Generic (PLEG): container finished" podID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerID="739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7" exitCode=0
Oct 30 20:42:07.982110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.982019    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"
Oct 30 20:42:07.982110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.982035    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" event={"ID":"74f553a9-eae7-44da-abf0-5f6181ea9656","Type":"ContainerDied","Data":"739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7"}
Oct 30 20:42:07.982110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.982096    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48" event={"ID":"74f553a9-eae7-44da-abf0-5f6181ea9656","Type":"ContainerDied","Data":"7a8d5d49aaffe45780c3297077778f365cbe332489f3f00e92b92373a1a7babe"}
Oct 30 20:42:07.982261 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.982118    5967 scope.go:122] "RemoveContainer" containerID="739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7"
Oct 30 20:42:07.993560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.993546    5967 scope.go:122] "RemoveContainer" containerID="739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7"
Oct 30 20:42:07.993841 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:42:07.993822    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7\": container with ID starting with 739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7 not found: ID does not exist" containerID="739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7"
Oct 30 20:42:07.993905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:07.993846    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7"} err="failed to get container status \"739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7\": rpc error: code = NotFound desc = could not find container \"739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7\": container with ID starting with 739271c523fa4c2198578ee7c991098a73840ffff409bd43d09ac66a67a596e7 not found: ID does not exist"
Oct 30 20:42:08.149272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:08.149245    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"]
Oct 30 20:42:08.156796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:08.156775    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48"]
Oct 30 20:42:08.313281 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:42:08.313250    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1fa104_3fab_4a1c_aff9_b286df34e1af.slice/crio-7001f19145338bf44cc7e053b91157db2cccc8bcb29b81b7f3d6831eb0f25b16 WatchSource:0}: Error finding container 7001f19145338bf44cc7e053b91157db2cccc8bcb29b81b7f3d6831eb0f25b16: Status 404 returned error can't find the container with id 7001f19145338bf44cc7e053b91157db2cccc8bcb29b81b7f3d6831eb0f25b16
Oct 30 20:42:08.314714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:08.314688    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"]
Oct 30 20:42:08.315436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:08.315421    5967 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:42:08.995526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:08.995485    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" event={"ID":"1f1fa104-3fab-4a1c-aff9-b286df34e1af","Type":"ContainerStarted","Data":"7001f19145338bf44cc7e053b91157db2cccc8bcb29b81b7f3d6831eb0f25b16"}
Oct 30 20:42:09.544068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:09.544033    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" path="/var/lib/kubelet/pods/74f553a9-eae7-44da-abf0-5f6181ea9656/volumes"
Oct 30 20:42:10.257033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.256999    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"]
Oct 30 20:42:10.258026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.258006    5967 cpu_manager.go:414] "containerMap: removing container" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy"
Oct 30 20:42:10.258026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.258027    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy"
Oct 30 20:42:10.258224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.258212    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f553a9-eae7-44da-abf0-5f6181ea9656" containerName="istio-proxy"
Oct 30 20:42:10.262488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.262466    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.320684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.320641    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"]
Oct 30 20:42:10.362024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.361992    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:42:10.362940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.362915    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:10.363076 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.362993    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:42:10.363656 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.363635    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:42:10.398299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.398269    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m99hw\" (UniqueName: \"kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.398446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.398344    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.398446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.398424    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.500238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.500198    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m99hw\" (UniqueName: \"kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.500390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.500281    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.500460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.500440    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.501928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.501115    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.504679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.504658    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.557182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.557133    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m99hw\" (UniqueName: \"kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw\") pod \"maas-api-revoke-sub-5c0d4bcf-tzqd9\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.580272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.580247    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:10.975725 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:10.975695    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"]
Oct 30 20:42:10.976183 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:42:10.976162    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147623a5_a309_4a72_a70e_88fe59456ef5.slice/crio-04c3b625c499b73a49fae4fe39707847f0dafd0d75bab3f4ce25e36cc6d238ba WatchSource:0}: Error finding container 04c3b625c499b73a49fae4fe39707847f0dafd0d75bab3f4ce25e36cc6d238ba: Status 404 returned error can't find the container with id 04c3b625c499b73a49fae4fe39707847f0dafd0d75bab3f4ce25e36cc6d238ba
Oct 30 20:42:11.014523 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:11.014487    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9" event={"ID":"147623a5-a309-4a72-a70e-88fe59456ef5","Type":"ContainerStarted","Data":"04c3b625c499b73a49fae4fe39707847f0dafd0d75bab3f4ce25e36cc6d238ba"}
Oct 30 20:42:11.961501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:11.961467    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:11.961797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:11.961534    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:12.025464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:12.025433    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9" event={"ID":"147623a5-a309-4a72-a70e-88fe59456ef5","Type":"ContainerStarted","Data":"d53c940d36d64712f52251290c19db6555b64a10044922545d034965711127dd"}
Oct 30 20:42:12.145560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:12.145516    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9" podStartSLOduration=2.14550287 podStartE2EDuration="2.14550287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:12.142016283 +0000 UTC m=+1312.742818242" watchObservedRunningTime="2026-10-30 20:42:12.14550287 +0000 UTC m=+1312.746304830"
Oct 30 20:42:13.039534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.039501    5967 generic.go:356] "Generic (PLEG): container finished" podID="147623a5-a309-4a72-a70e-88fe59456ef5" containerID="d53c940d36d64712f52251290c19db6555b64a10044922545d034965711127dd" exitCode=0
Oct 30 20:42:13.039998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.039591    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9" event={"ID":"147623a5-a309-4a72-a70e-88fe59456ef5","Type":"ContainerDied","Data":"d53c940d36d64712f52251290c19db6555b64a10044922545d034965711127dd"}
Oct 30 20:42:13.041447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.041422    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" event={"ID":"1f1fa104-3fab-4a1c-aff9-b286df34e1af","Type":"ContainerStarted","Data":"cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29"}
Oct 30 20:42:13.041626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.041609    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:13.043836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.043798    5967 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:42:13.043953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.043860    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" podUID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:42:13.299207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.299091    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" podStartSLOduration=2.654494825 podStartE2EDuration="6.299075737s" totalImagesPullingTime="3.644580912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:13.290842522 +0000 UTC m=+1313.891644488" watchObservedRunningTime="2026-10-30 20:42:13.299075737 +0000 UTC m=+1313.899877692"
Oct 30 20:42:13.946395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:13.946368    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:42:14.051084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.051061    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 20:42:14.238369 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.238344    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:14.359719 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.359690    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca\") pod \"147623a5-a309-4a72-a70e-88fe59456ef5\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") "
Oct 30 20:42:14.359924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.359842    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw\" (UniqueName: \"kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw\") pod \"147623a5-a309-4a72-a70e-88fe59456ef5\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") "
Oct 30 20:42:14.359988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.359974    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca\") pod \"147623a5-a309-4a72-a70e-88fe59456ef5\" (UID: \"147623a5-a309-4a72-a70e-88fe59456ef5\") "
Oct 30 20:42:14.360348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.360320    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca" pod "147623a5-a309-4a72-a70e-88fe59456ef5" (UID: "147623a5-a309-4a72-a70e-88fe59456ef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:14.360553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.360508    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:14.362148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.362119    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw" pod "147623a5-a309-4a72-a70e-88fe59456ef5" (UID: "147623a5-a309-4a72-a70e-88fe59456ef5"). InnerVolumeSpecName "kube-api-access-m99hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:14.362256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.362183    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca" pod "147623a5-a309-4a72-a70e-88fe59456ef5" (UID: "147623a5-a309-4a72-a70e-88fe59456ef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:14.461665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.461602    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147623a5-a309-4a72-a70e-88fe59456ef5-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:14.461665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.461632    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m99hw\" (UniqueName: \"kubernetes.io/projected/147623a5-a309-4a72-a70e-88fe59456ef5-kube-api-access-m99hw\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:14.957871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:14.957846    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:42:15.060622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:15.060597    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"
Oct 30 20:42:15.061063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:15.060636    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9" event={"ID":"147623a5-a309-4a72-a70e-88fe59456ef5","Type":"ContainerDied","Data":"04c3b625c499b73a49fae4fe39707847f0dafd0d75bab3f4ce25e36cc6d238ba"}
Oct 30 20:42:15.061063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:15.060686    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c3b625c499b73a49fae4fe39707847f0dafd0d75bab3f4ce25e36cc6d238ba"
Oct 30 20:42:19.792016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.791990    5967 scope.go:122] "RemoveContainer" containerID="776bac5333a5cb24dda9e6ed3c2e69bf66b27d428b18feb958eb06a8042ac23d"
Oct 30 20:42:19.802781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.802758    5967 scope.go:122] "RemoveContainer" containerID="f14153840d845c46994a786c0f584fdb94dda44076d69dd9e603b222bd7b49e3"
Oct 30 20:42:19.813160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.813141    5967 scope.go:122] "RemoveContainer" containerID="67df9b787646a8b2fc4ed3cd522d33d534758b60bbc9212e57d4e6051684c209"
Oct 30 20:42:19.823166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.823132    5967 scope.go:122] "RemoveContainer" containerID="154bde4ccd37137132e07804e684f7175007340a12e169f3ab8891ee71eb45b2"
Oct 30 20:42:19.836868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.836848    5967 scope.go:122] "RemoveContainer" containerID="98a9e06c6feead5ba0b4e6ddf8b518d2108b265154b977564b535598b4b910ec"
Oct 30 20:42:19.846221 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.846203    5967 scope.go:122] "RemoveContainer" containerID="cdd5e01aeba109f32b98830d4be2b4ae6fb617add709e462f4e45d465e47c2df"
Oct 30 20:42:19.855360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.855342    5967 scope.go:122] "RemoveContainer" containerID="5a1a5b3fb0b841b8fa043cde930930e6f20a11e63ea8b73fff322eb58496e250"
Oct 30 20:42:19.864764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.864728    5967 scope.go:122] "RemoveContainer" containerID="a90f8564297c382ee361ae545c4a929c7ad0e59e88172b4843bd06b1803f721f"
Oct 30 20:42:19.880159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.880139    5967 scope.go:122] "RemoveContainer" containerID="194000d985a2cfd9798c18af3246d14debf1ae5b18bc0aa2e392565506ce0d19"
Oct 30 20:42:19.889409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.889390    5967 scope.go:122] "RemoveContainer" containerID="0f6f552299075e20ea7dca66ff40873bf2d0565a6fb6c6923411d92efdc327de"
Oct 30 20:42:19.901754 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.901726    5967 scope.go:122] "RemoveContainer" containerID="d0bf882a178ea0e14ab10f480399dadc8c2aa0d7ca92b2166d03b1ab90719d46"
Oct 30 20:42:19.910599 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.910585    5967 scope.go:122] "RemoveContainer" containerID="831e2d1b9f257db8b6f0193668cc2306644c545ff1a0c2b3be0c6f9f836db42b"
Oct 30 20:42:19.919553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:19.919538    5967 scope.go:122] "RemoveContainer" containerID="8365a804326a9502576560b91ea607c2509fcc1d6676f222fc8e5229bccc9f75"
Oct 30 20:42:28.749398 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.749356    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"]
Oct 30 20:42:28.751767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.750095    5967 cpu_manager.go:414] "containerMap: removing container" podUID="147623a5-a309-4a72-a70e-88fe59456ef5" containerName="revoke-keys"
Oct 30 20:42:28.751767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.750109    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147623a5-a309-4a72-a70e-88fe59456ef5" containerName="revoke-keys"
Oct 30 20:42:28.751767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.750235    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="147623a5-a309-4a72-a70e-88fe59456ef5" containerName="revoke-keys"
Oct 30 20:42:28.753998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.753982    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.860583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.860548    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.860819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.860684    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.860819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.860807    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bb6b\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.860964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.860869    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.860964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.860936    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.861105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.860989    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.861217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.861191    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.861349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.861253    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.861349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.861287    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.870805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.870779    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"]
Oct 30 20:42:28.963089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963058    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963106    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963134    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963185    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963237    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bb6b\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963275    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963340    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963396    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963506    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963513    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963531    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963729    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.963845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.963805    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.964211 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.964194    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.965733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.965703    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:28.965835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:28.965785    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:29.020857 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.020787    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bb6b\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:29.023108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.023080    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kwkj5\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:29.068796 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.068771    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:29.516657 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:42:29.516627    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34803b24_bb46_4c76_86a9_2a7b646d9160.slice/crio-ee46934a231050c641ebbc767f1e503f584d208f8af052034cb535efb797faab WatchSource:0}: Error finding container ee46934a231050c641ebbc767f1e503f584d208f8af052034cb535efb797faab: Status 404 returned error can't find the container with id ee46934a231050c641ebbc767f1e503f584d208f8af052034cb535efb797faab
Oct 30 20:42:29.517265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.517224    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"]
Oct 30 20:42:29.521177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.519870    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:29.521177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.520019    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:29.521177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:29.520106    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:30.191067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:30.191029    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" event={"ID":"34803b24-bb46-4c76-86a9-2a7b646d9160","Type":"ContainerStarted","Data":"5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e"}
Oct 30 20:42:30.191450 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:30.191090    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" event={"ID":"34803b24-bb46-4c76-86a9-2a7b646d9160","Type":"ContainerStarted","Data":"ee46934a231050c641ebbc767f1e503f584d208f8af052034cb535efb797faab"}
Oct 30 20:42:30.420333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:30.420279    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podStartSLOduration=2.420264377 podStartE2EDuration="2.420264377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:30.369142363 +0000 UTC m=+1330.969944343" watchObservedRunningTime="2026-10-30 20:42:30.420264377 +0000 UTC m=+1331.021066336"
Oct 30 20:42:31.091777 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:31.090067    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:31.093981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:31.093233    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:31.093981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:31.093288    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:32.003812 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:42:32.003771    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:42:32.091766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:32.090055    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:32.091766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:32.090112    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:33.093880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:33.093841    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:33.094317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:33.093900    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:34.084770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:34.084022    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:34.084770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:34.084079    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:35.085768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:35.071044    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:35.085768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:35.071101    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:36.069558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:36.069527    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:36.069729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:36.069575    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:37.069424 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:37.069390    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:42:37.069967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:37.069439    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:42:38.106425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:38.106395    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:38.106956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:38.106783    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:38.107565 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:38.107548    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 20:42:43.844152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.844117    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"]
Oct 30 20:42:43.850652 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.850631    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.945716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.945681    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.945940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.945840    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.945941    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.945975    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.946011    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.946159    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nsb\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.946179    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.946195    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:43.946294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:43.946221    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047323    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nsb\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047363    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047569 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047473    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047569 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047521    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047596    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047647    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047758    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047814    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047862    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047938 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047916    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.047985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.047954    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.048163 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.048140    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.048263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.048244    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.048316 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.048301    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.050149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.050125    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.050383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.050364    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.094496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.094432    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"]
Oct 30 20:42:44.124083 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.124054    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.124378 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.124351    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nsb\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb\") pod \"maas-default-gateway-openshift-default-6886b96f9-pdvns\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.166468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.166445    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:44.606269 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:42:44.606238    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29698393_dfcb_4a5c_8d99_eba4d45f1aba.slice/crio-08df1aa9a7251492dd5b8da31697ebe2e29a4333b239397e1e73bde3f4ea93b0 WatchSource:0}: Error finding container 08df1aa9a7251492dd5b8da31697ebe2e29a4333b239397e1e73bde3f4ea93b0: Status 404 returned error can't find the container with id 08df1aa9a7251492dd5b8da31697ebe2e29a4333b239397e1e73bde3f4ea93b0
Oct 30 20:42:44.606389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.606294    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"]
Oct 30 20:42:44.608497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.608464    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.608591 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.608528    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:44.608591 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:44.608558    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:45.351202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:45.351167    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" event={"ID":"29698393-dfcb-4a5c-8d99-eba4d45f1aba","Type":"ContainerStarted","Data":"86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672"}
Oct 30 20:42:45.351615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:45.351229    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" event={"ID":"29698393-dfcb-4a5c-8d99-eba4d45f1aba","Type":"ContainerStarted","Data":"08df1aa9a7251492dd5b8da31697ebe2e29a4333b239397e1e73bde3f4ea93b0"}
Oct 30 20:42:45.488311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:45.488253    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podStartSLOduration=2.488235653 podStartE2EDuration="2.488235653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:45.488063136 +0000 UTC m=+1346.088865098" watchObservedRunningTime="2026-10-30 20:42:45.488235653 +0000 UTC m=+1346.089037614"
Oct 30 20:42:46.167686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:46.167654    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:46.191908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:46.191872    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:42:46.192253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:46.192220    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 20:42:47.167145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:47.167097    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:42:47.167621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:47.167158    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 20:42:48.166814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:48.166782    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:42:48.166985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:48.166831    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 20:42:49.167646 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:49.167614    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:42:49.168166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:49.167668    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 20:42:50.167557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:50.167519    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:42:50.167783 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:50.167572    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 20:42:51.168839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:51.168803    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:42:51.169249 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:51.168864    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 20:42:52.629940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.629901    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"]
Oct 30 20:42:52.639107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.639080    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.768358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.768329    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"]
Oct 30 20:42:52.799103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799060    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799115    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799211    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799323    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7tr\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799399    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799529    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799662    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799698    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.799770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.799733    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901135    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901167    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901195    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901257    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901297    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901319    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901406    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7tr\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901476    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901587    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901593    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901675    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.901937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.901859    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.902250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.902226    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.902566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.902443    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.904226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.904199    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.904462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.904438    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.958256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.958225    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:52.958616 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:52.958599    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7tr\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:53.038007 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.037965    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:53.038408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.038352    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:53.039485 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.039468    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 20:42:53.255413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.255383    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:53.694502 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:42:53.694470    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75348b55_bae3_4fc5_9e4d_cccef1a7e338.slice/crio-c402373e6878d2b47a7b182075e67bf20a9f4bfdb40e9c796c7f6ec6fe5dcfe3 WatchSource:0}: Error finding container c402373e6878d2b47a7b182075e67bf20a9f4bfdb40e9c796c7f6ec6fe5dcfe3: Status 404 returned error can't find the container with id c402373e6878d2b47a7b182075e67bf20a9f4bfdb40e9c796c7f6ec6fe5dcfe3
Oct 30 20:42:53.695386 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.695342    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"]
Oct 30 20:42:53.697221 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.697185    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:53.697340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.697271    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:53.697340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:53.697317    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:54.460565 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:54.460533    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" event={"ID":"75348b55-bae3-4fc5-9e4d-cccef1a7e338","Type":"ContainerStarted","Data":"9a4a1670eef0e03637c663132f2180221f156d59adf3627a16ec92fedb217a24"}
Oct 30 20:42:54.460787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:54.460589    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" event={"ID":"75348b55-bae3-4fc5-9e4d-cccef1a7e338","Type":"ContainerStarted","Data":"c402373e6878d2b47a7b182075e67bf20a9f4bfdb40e9c796c7f6ec6fe5dcfe3"}
Oct 30 20:42:54.580447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:54.580389    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podStartSLOduration=2.580370671 podStartE2EDuration="2.580370671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:54.577176675 +0000 UTC m=+1355.177978635" watchObservedRunningTime="2026-10-30 20:42:54.580370671 +0000 UTC m=+1355.181172631"
Oct 30 20:42:55.255546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:55.255512    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:42:55.257239 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:55.257215    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:42:55.257364 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:55.257258    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:42:56.256271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:56.256239    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:42:56.256670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:56.256288    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:42:57.256407 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:57.256376    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:42:57.256947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:57.256426    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:42:58.256281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:58.256253    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:42:58.256463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:58.256301    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:42:59.256298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:59.256261    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:42:59.256498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:42:59.256325    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:43:00.255639 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:00.255604    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:43:00.255875 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:00.255671    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:43:01.256148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:01.256114    5967 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:43:01.256645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:01.256177    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:43:01.981214 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:43:01.981183    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:43:02.260504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:02.260423    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:43:02.260941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:02.260922    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:43:02.262066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:02.262041    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:43:14.253922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.253841    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:43:14.253922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.253904    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:43:14.703276 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.703242    5967 generic.go:356] "Generic (PLEG): container finished" podID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerID="8e08bdb41706159fbf1e29d725bc613db7c4f42c90b71306c72500f2e99e05e3" exitCode=0
Oct 30 20:43:14.704351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.703308    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" event={"ID":"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b","Type":"ContainerDied","Data":"8e08bdb41706159fbf1e29d725bc613db7c4f42c90b71306c72500f2e99e05e3"}
Oct 30 20:43:14.704351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.703685    5967 scope.go:122] "RemoveContainer" containerID="8e08bdb41706159fbf1e29d725bc613db7c4f42c90b71306c72500f2e99e05e3"
Oct 30 20:43:14.705125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.705097    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:14.705204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.705161    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:14.705204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:14.705194    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:15.713453 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.713419    5967 generic.go:356] "Generic (PLEG): container finished" podID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerID="8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad" exitCode=0
Oct 30 20:43:15.713942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.713487    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" event={"ID":"3903830d-6c6a-4f62-9f1a-9ed15aea1d41","Type":"ContainerDied","Data":"8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"}
Oct 30 20:43:15.714017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.713957    5967 scope.go:122] "RemoveContainer" containerID="8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"
Oct 30 20:43:15.715763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.715719    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:15.715881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.715798    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:15.715881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.715834    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:15.715973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.715940    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" event={"ID":"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b","Type":"ContainerStarted","Data":"32392bab6706f2cd69567a6abca13383104b0c984746fb4b11a153957ff5112f"}
Oct 30 20:43:15.858125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.858102    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:43:15.859003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.858978    5967 patch_prober.go:29] interesting pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:43:15.859101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:15.859039    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 20:43:16.731911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:16.731882    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" event={"ID":"3903830d-6c6a-4f62-9f1a-9ed15aea1d41","Type":"ContainerStarted","Data":"64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef"}
Oct 30 20:43:16.857630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:16.857596    5967 patch_prober.go:29] interesting pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:43:16.857834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:16.857654    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 20:43:17.338106 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:17.338074    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:43:17.338273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:17.338246    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:43:17.338317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:17.338284    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:43:17.858335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:17.858299    5967 patch_prober.go:29] interesting pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:43:17.858880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:17.858348    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 20:43:18.290767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:18.287849    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:43:18.337948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:18.337920    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:43:18.338170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:18.338150    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:43:18.857726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:18.857695    5967 patch_prober.go:29] interesting pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:43:18.857935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:18.857780    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 20:43:19.338119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:19.338085    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:43:19.338530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:19.338137    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:43:19.857602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:19.857564    5967 patch_prober.go:29] interesting pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:43:19.857896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:19.857614    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 20:43:20.020389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.020341    5967 scope.go:122] "RemoveContainer" containerID="df4b875c9a93f8485b8c9cc89c3259451681069d2ccb8e23b45e767d3d54d513"
Oct 30 20:43:20.031884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.031868    5967 scope.go:122] "RemoveContainer" containerID="235c8ff4a3399ac09f86176ccce684765d7ab2ce155afe8257a2947f1e58a6b0"
Oct 30 20:43:20.042248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.042230    5967 scope.go:122] "RemoveContainer" containerID="c69086a28b814f22971d950830b319e256ae27f030bc94623fe265fe18d7a0f2"
Oct 30 20:43:20.052312 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.052295    5967 scope.go:122] "RemoveContainer" containerID="85d6597afe0039229ba24943d1870a7130d859945aee7e06924a21ef82879908"
Oct 30 20:43:20.063291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.063274    5967 scope.go:122] "RemoveContainer" containerID="1ab5daef454ee0796d0c47a7960d0b3535d54a0a672336c9f13859fad945d209"
Oct 30 20:43:20.122633 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.122610    5967 scope.go:122] "RemoveContainer" containerID="f59f5b05f87c0c408851927a20458c620f8ca78d9ca10de47bd288edeb13d184"
Oct 30 20:43:20.338278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.338247    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:43:20.338651 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.338294    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:43:20.858118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.858083    5967 patch_prober.go:29] interesting pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:43:20.858299 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:20.858147    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 20:43:21.338460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:21.338426    5967 patch_prober.go:29] interesting pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:43:21.338908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:21.338479    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:43:21.863290 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:21.863260    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:43:21.865220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:21.865193    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:43:22.341708 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:22.341678    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:43:22.342149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:22.342130    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:43:22.342811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:22.342795    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:43:31.982124 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:43:31.982090    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:43:36.184940 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.184898    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"]
Oct 30 20:43:36.189702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.189686    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.245858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.245830    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"]
Oct 30 20:43:36.289603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.289569    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:43:36.289752 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.289635    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:43:36.290721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.290705    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:43:36.290798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.290704    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:43:36.310145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.310113    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.310305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.310249    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsd6\" (UniqueName: \"kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.310346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.310334    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.411407 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.411377    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.411582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.411473    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsd6\" (UniqueName: \"kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.411582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.411569    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.412139 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.412113    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.414108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.414088    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.468465 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.468432    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsd6\" (UniqueName: \"kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6\") pod \"maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.503394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.503372    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:36.881552 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:43:36.881510    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf605daed_4083_46e6_8dbc_a68d968268b2.slice/crio-2e50f7a9b2f02f45c403d15d717c359a0b3132ae78de8e3d9efec736dd782f72 WatchSource:0}: Error finding container 2e50f7a9b2f02f45c403d15d717c359a0b3132ae78de8e3d9efec736dd782f72: Status 404 returned error can't find the container with id 2e50f7a9b2f02f45c403d15d717c359a0b3132ae78de8e3d9efec736dd782f72
Oct 30 20:43:36.883362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.883341    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"]
Oct 30 20:43:36.909886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:36.909856    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq" event={"ID":"f605daed-4083-46e6-8dbc-a68d968268b2","Type":"ContainerStarted","Data":"2e50f7a9b2f02f45c403d15d717c359a0b3132ae78de8e3d9efec736dd782f72"}
Oct 30 20:43:37.919530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:37.919496    5967 generic.go:356] "Generic (PLEG): container finished" podID="f605daed-4083-46e6-8dbc-a68d968268b2" containerID="91d7bf4215bf2685a5a9cb268ac91833edd355849b8642516a31c1b0f6b3afca" exitCode=0
Oct 30 20:43:37.919951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:37.919584    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq" event={"ID":"f605daed-4083-46e6-8dbc-a68d968268b2","Type":"ContainerDied","Data":"91d7bf4215bf2685a5a9cb268ac91833edd355849b8642516a31c1b0f6b3afca"}
Oct 30 20:43:38.291781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.291725    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"]
Oct 30 20:43:38.295358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.295337    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.355084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.355057    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"]
Oct 30 20:43:38.442272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.442242    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.442436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.442414    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.442491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.442477    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pll8\" (UniqueName: \"kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.543673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.543595    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.543673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.543647    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pll8\" (UniqueName: \"kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.543908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.543827    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.544477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.544400    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.546111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.546092    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.602449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.602425    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pll8\" (UniqueName: \"kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8\") pod \"maas-api-revoke-keys-e2e-ait-8660b337-5xnpm\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.608266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.608247    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:38.992509 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:38.987339    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"]
Oct 30 20:43:39.190334 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.190312    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:39.254505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.254476    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6\" (UniqueName: \"kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6\") pod \"f605daed-4083-46e6-8dbc-a68d968268b2\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") "
Oct 30 20:43:39.254685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.254539    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca\") pod \"f605daed-4083-46e6-8dbc-a68d968268b2\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") "
Oct 30 20:43:39.254685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.254564    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca\") pod \"f605daed-4083-46e6-8dbc-a68d968268b2\" (UID: \"f605daed-4083-46e6-8dbc-a68d968268b2\") "
Oct 30 20:43:39.254990 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.254959    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca" pod "f605daed-4083-46e6-8dbc-a68d968268b2" (UID: "f605daed-4083-46e6-8dbc-a68d968268b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:39.255294 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.255279    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:39.256757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.256714    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca" pod "f605daed-4083-46e6-8dbc-a68d968268b2" (UID: "f605daed-4083-46e6-8dbc-a68d968268b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:43:39.256925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.256908    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6" pod "f605daed-4083-46e6-8dbc-a68d968268b2" (UID: "f605daed-4083-46e6-8dbc-a68d968268b2"). InnerVolumeSpecName "kube-api-access-rnsd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:39.356709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.356677    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f605daed-4083-46e6-8dbc-a68d968268b2-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:39.356709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.356706    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsd6\" (UniqueName: \"kubernetes.io/projected/f605daed-4083-46e6-8dbc-a68d968268b2-kube-api-access-rnsd6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:39.936999 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.936971    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"
Oct 30 20:43:39.937170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.936998    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq" event={"ID":"f605daed-4083-46e6-8dbc-a68d968268b2","Type":"ContainerDied","Data":"2e50f7a9b2f02f45c403d15d717c359a0b3132ae78de8e3d9efec736dd782f72"}
Oct 30 20:43:39.937170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.937041    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e50f7a9b2f02f45c403d15d717c359a0b3132ae78de8e3d9efec736dd782f72"
Oct 30 20:43:39.939358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.939336    5967 generic.go:356] "Generic (PLEG): container finished" podID="d67a678c-8641-4f17-a6be-2c1c36d519ac" containerID="6dd51ea01f6e4ff609c7a04d8b877f5851b3241a0ca2b407970d9c0510f37ab4" exitCode=0
Oct 30 20:43:39.939480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.939422    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm" event={"ID":"d67a678c-8641-4f17-a6be-2c1c36d519ac","Type":"ContainerDied","Data":"6dd51ea01f6e4ff609c7a04d8b877f5851b3241a0ca2b407970d9c0510f37ab4"}
Oct 30 20:43:39.939480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:39.939459    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm" event={"ID":"d67a678c-8641-4f17-a6be-2c1c36d519ac","Type":"ContainerStarted","Data":"dc1eafd34cfa76e32faa8e5027a2b3a0fd300dd644aa26432d333c8e0f71e8d1"}
Oct 30 20:43:41.116879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.116854    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:41.181159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.181126    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca\") pod \"d67a678c-8641-4f17-a6be-2c1c36d519ac\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") "
Oct 30 20:43:41.181297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.181232    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8\" (UniqueName: \"kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8\") pod \"d67a678c-8641-4f17-a6be-2c1c36d519ac\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") "
Oct 30 20:43:41.181366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.181334    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca\") pod \"d67a678c-8641-4f17-a6be-2c1c36d519ac\" (UID: \"d67a678c-8641-4f17-a6be-2c1c36d519ac\") "
Oct 30 20:43:41.181450 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.181429    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca" pod "d67a678c-8641-4f17-a6be-2c1c36d519ac" (UID: "d67a678c-8641-4f17-a6be-2c1c36d519ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:41.181734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.181718    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:41.183437 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.183414    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca" pod "d67a678c-8641-4f17-a6be-2c1c36d519ac" (UID: "d67a678c-8641-4f17-a6be-2c1c36d519ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:43:41.183518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.183505    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8" pod "d67a678c-8641-4f17-a6be-2c1c36d519ac" (UID: "d67a678c-8641-4f17-a6be-2c1c36d519ac"). InnerVolumeSpecName "kube-api-access-5pll8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:41.283278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.283248    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pll8\" (UniqueName: \"kubernetes.io/projected/d67a678c-8641-4f17-a6be-2c1c36d519ac-kube-api-access-5pll8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:41.283278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.283277    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67a678c-8641-4f17-a6be-2c1c36d519ac-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:41.956234 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.956161    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm" event={"ID":"d67a678c-8641-4f17-a6be-2c1c36d519ac","Type":"ContainerDied","Data":"dc1eafd34cfa76e32faa8e5027a2b3a0fd300dd644aa26432d333c8e0f71e8d1"}
Oct 30 20:43:41.956234 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.956193    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"
Oct 30 20:43:41.956234 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:41.956204    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc1eafd34cfa76e32faa8e5027a2b3a0fd300dd644aa26432d333c8e0f71e8d1"
Oct 30 20:43:48.381791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.381758    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"]
Oct 30 20:43:48.382491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.382476    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f605daed-4083-46e6-8dbc-a68d968268b2" containerName="revoke-keys"
Oct 30 20:43:48.382538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.382492    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f605daed-4083-46e6-8dbc-a68d968268b2" containerName="revoke-keys"
Oct 30 20:43:48.382538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.382530    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d67a678c-8641-4f17-a6be-2c1c36d519ac" containerName="revoke-keys"
Oct 30 20:43:48.382538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.382535    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67a678c-8641-4f17-a6be-2c1c36d519ac" containerName="revoke-keys"
Oct 30 20:43:48.382665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.382655    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67a678c-8641-4f17-a6be-2c1c36d519ac" containerName="revoke-keys"
Oct 30 20:43:48.382699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.382677    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f605daed-4083-46e6-8dbc-a68d968268b2" containerName="revoke-keys"
Oct 30 20:43:48.386726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.386710    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.392483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.392455    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.392594 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.392503    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qfb\" (UniqueName: \"kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.392639 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.392595    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.442912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.442886    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"]
Oct 30 20:43:48.485851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.485823    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:43:48.486013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.485880    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:43:48.487636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.487620    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:43:48.487685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.487620    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:43:48.494019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.493991    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.494165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.494039    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qfb\" (UniqueName: \"kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.494165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.494095    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.494733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.494709    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.496604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.496586    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.550911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.550884    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qfb\" (UniqueName: \"kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb\") pod \"maas-api-revoke-keys-e2e-ait-23cde229-vdxms\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:48.702461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:48.702397    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:49.145954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:49.145870    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"]
Oct 30 20:43:50.049039 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:50.049002    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms" event={"ID":"8649efd8-d749-4b51-bcc7-2356ff62e156","Type":"ContainerStarted","Data":"f72db77776e2b95e045006e00a37f1269dce355f52c5772361bf7235131bb87e"}
Oct 30 20:43:50.049439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:50.049066    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms" event={"ID":"8649efd8-d749-4b51-bcc7-2356ff62e156","Type":"ContainerStarted","Data":"4e7ca33f0d7a0d22be2cb559700d6aa3cbfce62fdaa286fbb3a3467f235be382"}
Oct 30 20:43:50.167732 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:50.167682    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms" podStartSLOduration=2.16766937 podStartE2EDuration="2.16766937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:50.16399727 +0000 UTC m=+1410.764799229" watchObservedRunningTime="2026-10-30 20:43:50.16766937 +0000 UTC m=+1410.768471329"
Oct 30 20:43:50.219820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:50.219788    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"]
Oct 30 20:43:51.058817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.058780    5967 generic.go:356] "Generic (PLEG): container finished" podID="8649efd8-d749-4b51-bcc7-2356ff62e156" containerID="f72db77776e2b95e045006e00a37f1269dce355f52c5772361bf7235131bb87e" exitCode=0
Oct 30 20:43:51.058817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.058820    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms" event={"ID":"8649efd8-d749-4b51-bcc7-2356ff62e156","Type":"ContainerDied","Data":"f72db77776e2b95e045006e00a37f1269dce355f52c5772361bf7235131bb87e"}
Oct 30 20:43:51.760307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.760185    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"]
Oct 30 20:43:51.765148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.765125    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.795349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.795312    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.795547 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.795380    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.795547 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.795440    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4mj9\" (UniqueName: \"kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.834125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.833722    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"]
Oct 30 20:43:51.914152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.906945    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.914482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.914454    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.920886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.914646    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4mj9\" (UniqueName: \"kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.921528 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.908910    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.945048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.945013    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:51.978622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:51.978590    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4mj9\" (UniqueName: \"kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9\") pod \"maas-api-revoke-sub-0c08e0c4-7thrx\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:52.089160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.089131    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:52.251286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.251252    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:52.327126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.327051    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb\" (UniqueName: \"kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb\") pod \"8649efd8-d749-4b51-bcc7-2356ff62e156\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") "
Oct 30 20:43:52.327126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.327095    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca\") pod \"8649efd8-d749-4b51-bcc7-2356ff62e156\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") "
Oct 30 20:43:52.327375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.327260    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca\") pod \"8649efd8-d749-4b51-bcc7-2356ff62e156\" (UID: \"8649efd8-d749-4b51-bcc7-2356ff62e156\") "
Oct 30 20:43:52.327505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.327483    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca" pod "8649efd8-d749-4b51-bcc7-2356ff62e156" (UID: "8649efd8-d749-4b51-bcc7-2356ff62e156"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:52.327981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.327961    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:52.329326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.329307    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb" pod "8649efd8-d749-4b51-bcc7-2356ff62e156" (UID: "8649efd8-d749-4b51-bcc7-2356ff62e156"). InnerVolumeSpecName "kube-api-access-k7qfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:52.329418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.329349    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca" pod "8649efd8-d749-4b51-bcc7-2356ff62e156" (UID: "8649efd8-d749-4b51-bcc7-2356ff62e156"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:43:52.429618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.429586    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qfb\" (UniqueName: \"kubernetes.io/projected/8649efd8-d749-4b51-bcc7-2356ff62e156-kube-api-access-k7qfb\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:52.429618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.429613    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8649efd8-d749-4b51-bcc7-2356ff62e156-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:52.481476 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:43:52.481449    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044de5a6_9abd_4ea5_89ff_720297490928.slice/crio-23c5fa6ea6b41ff7597222bb35bb731fb867d142e4992d693c9d4fb3ff7afbf9 WatchSource:0}: Error finding container 23c5fa6ea6b41ff7597222bb35bb731fb867d142e4992d693c9d4fb3ff7afbf9: Status 404 returned error can't find the container with id 23c5fa6ea6b41ff7597222bb35bb731fb867d142e4992d693c9d4fb3ff7afbf9
Oct 30 20:43:52.482377 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:52.482354    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"]
Oct 30 20:43:53.076688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:53.076658    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms" event={"ID":"8649efd8-d749-4b51-bcc7-2356ff62e156","Type":"ContainerDied","Data":"4e7ca33f0d7a0d22be2cb559700d6aa3cbfce62fdaa286fbb3a3467f235be382"}
Oct 30 20:43:53.076894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:53.076706    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7ca33f0d7a0d22be2cb559700d6aa3cbfce62fdaa286fbb3a3467f235be382"
Oct 30 20:43:53.076894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:53.076671    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"
Oct 30 20:43:53.078583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:53.078560    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx" event={"ID":"044de5a6-9abd-4ea5-89ff-720297490928","Type":"ContainerStarted","Data":"279e622353f29c45b76084b4289d332ec3de3220188f5f1c39ab04f685ec04da"}
Oct 30 20:43:53.078680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:53.078606    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx" event={"ID":"044de5a6-9abd-4ea5-89ff-720297490928","Type":"ContainerStarted","Data":"23c5fa6ea6b41ff7597222bb35bb731fb867d142e4992d693c9d4fb3ff7afbf9"}
Oct 30 20:43:53.192615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:53.192562    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx" podStartSLOduration=2.192548186 podStartE2EDuration="2.192548186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:53.191813774 +0000 UTC m=+1413.792615733" watchObservedRunningTime="2026-10-30 20:43:53.192548186 +0000 UTC m=+1413.793350144"
Oct 30 20:43:54.088264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:54.088228    5967 generic.go:356] "Generic (PLEG): container finished" podID="044de5a6-9abd-4ea5-89ff-720297490928" containerID="279e622353f29c45b76084b4289d332ec3de3220188f5f1c39ab04f685ec04da" exitCode=0
Oct 30 20:43:54.088436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:54.088315    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx" event={"ID":"044de5a6-9abd-4ea5-89ff-720297490928","Type":"ContainerDied","Data":"279e622353f29c45b76084b4289d332ec3de3220188f5f1c39ab04f685ec04da"}
Oct 30 20:43:55.220281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.220222    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" podUID="f29a78b7-1351-4fe0-bcca-29ed0a717347" containerName="payload-pre-processing" containerID="cri-o://31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7" gracePeriod=25
Oct 30 20:43:55.357472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.357406    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:55.469671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.469645    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9\" (UniqueName: \"kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9\") pod \"044de5a6-9abd-4ea5-89ff-720297490928\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") "
Oct 30 20:43:55.471460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.469880    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca\") pod \"044de5a6-9abd-4ea5-89ff-720297490928\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") "
Oct 30 20:43:55.471460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.470121    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca\") pod \"044de5a6-9abd-4ea5-89ff-720297490928\" (UID: \"044de5a6-9abd-4ea5-89ff-720297490928\") "
Oct 30 20:43:55.471460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.470503    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca" pod "044de5a6-9abd-4ea5-89ff-720297490928" (UID: "044de5a6-9abd-4ea5-89ff-720297490928"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:55.471460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.470846    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:55.480727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.480701    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca" pod "044de5a6-9abd-4ea5-89ff-720297490928" (UID: "044de5a6-9abd-4ea5-89ff-720297490928"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:43:55.480959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.480940    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9" pod "044de5a6-9abd-4ea5-89ff-720297490928" (UID: "044de5a6-9abd-4ea5-89ff-720297490928"). InnerVolumeSpecName "kube-api-access-j4mj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:55.506151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.506128    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:43:55.571447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.571416    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j\" (UniqueName: \"kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j\") pod \"f29a78b7-1351-4fe0-bcca-29ed0a717347\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") "
Oct 30 20:43:55.571580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.571550    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume\") pod \"f29a78b7-1351-4fe0-bcca-29ed0a717347\" (UID: \"f29a78b7-1351-4fe0-bcca-29ed0a717347\") "
Oct 30 20:43:55.571912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.571889    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume" pod "f29a78b7-1351-4fe0-bcca-29ed0a717347" (UID: "f29a78b7-1351-4fe0-bcca-29ed0a717347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:43:55.572325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.572305    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044de5a6-9abd-4ea5-89ff-720297490928-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:55.572435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.572375    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29a78b7-1351-4fe0-bcca-29ed0a717347-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:55.572435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.572390    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4mj9\" (UniqueName: \"kubernetes.io/projected/044de5a6-9abd-4ea5-89ff-720297490928-kube-api-access-j4mj9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:55.573588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.573572    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j" pod "f29a78b7-1351-4fe0-bcca-29ed0a717347" (UID: "f29a78b7-1351-4fe0-bcca-29ed0a717347"). InnerVolumeSpecName "kube-api-access-h8l2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:43:55.673834 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:55.673767    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8l2j\" (UniqueName: \"kubernetes.io/projected/f29a78b7-1351-4fe0-bcca-29ed0a717347-kube-api-access-h8l2j\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:43:56.123805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.123729    5967 generic.go:356] "Generic (PLEG): container finished" podID="f29a78b7-1351-4fe0-bcca-29ed0a717347" containerID="31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7" exitCode=0
Oct 30 20:43:56.123991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.123823    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"
Oct 30 20:43:56.123991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.123782    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" event={"ID":"f29a78b7-1351-4fe0-bcca-29ed0a717347","Type":"ContainerDied","Data":"31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7"}
Oct 30 20:43:56.123991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.123905    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" event={"ID":"f29a78b7-1351-4fe0-bcca-29ed0a717347","Type":"ContainerDied","Data":"51ef966a8853ab67b6b3fad65f6a364b88395277c17ac3384571210c76241e76"}
Oct 30 20:43:56.123991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.123934    5967 scope.go:122] "RemoveContainer" containerID="31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7"
Oct 30 20:43:56.126348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.125791    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"
Oct 30 20:43:56.126348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.125817    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx" event={"ID":"044de5a6-9abd-4ea5-89ff-720297490928","Type":"ContainerDied","Data":"23c5fa6ea6b41ff7597222bb35bb731fb867d142e4992d693c9d4fb3ff7afbf9"}
Oct 30 20:43:56.126348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.125857    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c5fa6ea6b41ff7597222bb35bb731fb867d142e4992d693c9d4fb3ff7afbf9"
Oct 30 20:43:56.139881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.139856    5967 scope.go:122] "RemoveContainer" containerID="31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7"
Oct 30 20:43:56.140260 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:43:56.140229    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7\": container with ID starting with 31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7 not found: ID does not exist" containerID="31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7"
Oct 30 20:43:56.140391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.140272    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7"} err="failed to get container status \"31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7\": rpc error: code = NotFound desc = could not find container \"31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7\": container with ID starting with 31304112f07dd98430282579869a067b87c0f99e2032a48a146eb02c8c2d09f7 not found: ID does not exist"
Oct 30 20:43:56.344601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.344570    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"]
Oct 30 20:43:56.349222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.349200    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p"]
Oct 30 20:43:56.435780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:56.435648    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p" podUID="f29a78b7-1351-4fe0-bcca-29ed0a717347" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.133:50052\" within 1s: context deadline exceeded"
Oct 30 20:43:56.435952 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:43:56.435804    5967 logging.go:55] [core] [Channel #2615 SubChannel #2616]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.133:50052", ServerName: "10.134.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.133:50052: operation was canceled"
Oct 30 20:43:57.544336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:57.544303    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29a78b7-1351-4fe0-bcca-29ed0a717347" path="/var/lib/kubelet/pods/f29a78b7-1351-4fe0-bcca-29ed0a717347/volumes"
Oct 30 20:43:59.897109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:43:59.897069    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"]
Oct 30 20:44:00.022771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:00.021412    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"]
Oct 30 20:44:01.980969 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:44:01.980935    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:44:04.897838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:04.897778    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" podUID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" containerName="payload-processing" containerID="cri-o://6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba" gracePeriod=25
Oct 30 20:44:04.955467 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:44:04.955444    5967 logging.go:55] [core] [Channel #2653 SubChannel #2654]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: connect: connection refused"
Oct 30 20:44:05.022110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.022069    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr" podUID="ddd13fe6-31fc-4690-92cc-bea3ad21430a" containerName="payload-pre-processing" containerID="cri-o://82c3b345f8492fec702395f8b345791a1f6db69b933aae0fd9a99fb18a35e404" gracePeriod=25
Oct 30 20:44:05.174340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.174308    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:44:05.207592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.207552    5967 generic.go:356] "Generic (PLEG): container finished" podID="ddd13fe6-31fc-4690-92cc-bea3ad21430a" containerID="82c3b345f8492fec702395f8b345791a1f6db69b933aae0fd9a99fb18a35e404" exitCode=0
Oct 30 20:44:05.207729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.207614    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr" event={"ID":"ddd13fe6-31fc-4690-92cc-bea3ad21430a","Type":"ContainerDied","Data":"82c3b345f8492fec702395f8b345791a1f6db69b933aae0fd9a99fb18a35e404"}
Oct 30 20:44:05.209329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.209304    5967 generic.go:356] "Generic (PLEG): container finished" podID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" containerID="6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba" exitCode=0
Oct 30 20:44:05.209455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.209350    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"
Oct 30 20:44:05.209455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.209381    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" event={"ID":"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd","Type":"ContainerDied","Data":"6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba"}
Oct 30 20:44:05.209455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.209431    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" event={"ID":"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd","Type":"ContainerDied","Data":"28051f80a267877501e12e4cbda45b4af7d8f0ee85c10730ea581af8516c5b17"}
Oct 30 20:44:05.209561 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.209460    5967 scope.go:122] "RemoveContainer" containerID="6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba"
Oct 30 20:44:05.221283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.221258    5967 scope.go:122] "RemoveContainer" containerID="6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba"
Oct 30 20:44:05.222420 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:44:05.222383    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba\": container with ID starting with 6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba not found: ID does not exist" containerID="6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba"
Oct 30 20:44:05.222541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.222422    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba"} err="failed to get container status \"6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba\": rpc error: code = NotFound desc = could not find container \"6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba\": container with ID starting with 6fe19ce59a8111b4ece7dd3b4c2d48c34e519ace8c836ee10e9bc9f72752f1ba not found: ID does not exist"
Oct 30 20:44:05.263418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.262973    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx\" (UniqueName: \"kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx\") pod \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") "
Oct 30 20:44:05.263418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.263112    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume\") pod \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\" (UID: \"a7b6cddc-f1f5-41a1-8775-a74b0386bfdd\") "
Oct 30 20:44:05.264371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.264348    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume" pod "a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" (UID: "a7b6cddc-f1f5-41a1-8775-a74b0386bfdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:05.266482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.266455    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx" pod "a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" (UID: "a7b6cddc-f1f5-41a1-8775-a74b0386bfdd"). InnerVolumeSpecName "kube-api-access-x5qrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:05.288908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.288887    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:44:05.365131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.365107    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume\") pod \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") "
Oct 30 20:44:05.365281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.365155    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg\" (UniqueName: \"kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg\") pod \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\" (UID: \"ddd13fe6-31fc-4690-92cc-bea3ad21430a\") "
Oct 30 20:44:05.365447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.365426    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume" pod "ddd13fe6-31fc-4690-92cc-bea3ad21430a" (UID: "ddd13fe6-31fc-4690-92cc-bea3ad21430a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:05.365799 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.365760    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd13fe6-31fc-4690-92cc-bea3ad21430a-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:05.365799 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.365785    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5qrx\" (UniqueName: \"kubernetes.io/projected/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-kube-api-access-x5qrx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:05.365799 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.365798    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:05.367380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.367362    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg" pod "ddd13fe6-31fc-4690-92cc-bea3ad21430a" (UID: "ddd13fe6-31fc-4690-92cc-bea3ad21430a"). InnerVolumeSpecName "kube-api-access-447zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:05.467617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.467585    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447zg\" (UniqueName: \"kubernetes.io/projected/ddd13fe6-31fc-4690-92cc-bea3ad21430a-kube-api-access-447zg\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:05.641852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.641810    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"]
Oct 30 20:44:05.644989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.644962    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x"]
Oct 30 20:44:05.955222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:05.955192    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x" podUID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Oct 30 20:44:06.218833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:06.218796    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr" event={"ID":"ddd13fe6-31fc-4690-92cc-bea3ad21430a","Type":"ContainerDied","Data":"1851e8701eadfc421b3d528cde052be116840f7fc8bd9eb48c89e91a38b741bc"}
Oct 30 20:44:06.219017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:06.218808    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"
Oct 30 20:44:06.219017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:06.218895    5967 scope.go:122] "RemoveContainer" containerID="82c3b345f8492fec702395f8b345791a1f6db69b933aae0fd9a99fb18a35e404"
Oct 30 20:44:06.386713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:06.386674    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"]
Oct 30 20:44:06.391125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:06.391106    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr"]
Oct 30 20:44:07.544623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:07.544590    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" path="/var/lib/kubelet/pods/a7b6cddc-f1f5-41a1-8775-a74b0386bfdd/volumes"
Oct 30 20:44:07.545055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:07.545040    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd13fe6-31fc-4690-92cc-bea3ad21430a" path="/var/lib/kubelet/pods/ddd13fe6-31fc-4690-92cc-bea3ad21430a/volumes"
Oct 30 20:44:21.611560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:21.611498    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"]
Oct 30 20:44:21.656921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:21.656892    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f0ceebd-vs4hq"]
Oct 30 20:44:22.874625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:22.874593    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"]
Oct 30 20:44:22.924977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:22.924940    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8660b337-5xnpm"]
Oct 30 20:44:23.467176 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:23.467139    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"]
Oct 30 20:44:23.467651 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:23.467620    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy" containerID="cri-o://64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef" gracePeriod=30
Oct 30 20:44:23.545176 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:23.545145    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67a678c-8641-4f17-a6be-2c1c36d519ac" path="/var/lib/kubelet/pods/d67a678c-8641-4f17-a6be-2c1c36d519ac/volumes"
Oct 30 20:44:23.545645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:23.545627    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f605daed-4083-46e6-8dbc-a68d968268b2" path="/var/lib/kubelet/pods/f605daed-4083-46e6-8dbc-a68d968268b2/volumes"
Oct 30 20:44:25.071943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:25.071901    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"]
Oct 30 20:44:25.119816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:25.119786    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59f45b-tjmg6"]
Oct 30 20:44:25.545392 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:25.545362    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e" path="/var/lib/kubelet/pods/2dd387b4-4fa9-46f1-9e40-0c280cbe2a5e/volumes"
Oct 30 20:44:28.927216 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:28.927192    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:44:29.024439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024408    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.024599 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024511    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.024599 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024578    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.024714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024632    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.024714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024636    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:29.024714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024693    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.024943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024801    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.024943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024834    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:29.024943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024878    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:29.024943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024917    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.025144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.024954    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.025144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025003    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data\") pod \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\" (UID: \"3903830d-6c6a-4f62-9f1a-9ed15aea1d41\") "
Oct 30 20:44:29.025338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025318    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:29.025399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025358    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:29.025894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025875    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.025963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025901    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.025963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025918    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.025963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025932    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.025963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.025946    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.027156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.027136    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:44:29.027225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.027181    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:29.027443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.027427    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "kube-api-access-vfx7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:29.027487 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.027438    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy" pod "3903830d-6c6a-4f62-9f1a-9ed15aea1d41" (UID: "3903830d-6c6a-4f62-9f1a-9ed15aea1d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:29.127681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.127607    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfx7x\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-kube-api-access-vfx7x\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.127681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.127635    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.127681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.127645    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.127681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.127655    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3903830d-6c6a-4f62-9f1a-9ed15aea1d41-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.449220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.449128    5967 generic.go:356] "Generic (PLEG): container finished" podID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerID="64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef" exitCode=0
Oct 30 20:44:29.449373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.449226    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"
Oct 30 20:44:29.449373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.449227    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" event={"ID":"3903830d-6c6a-4f62-9f1a-9ed15aea1d41","Type":"ContainerDied","Data":"64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef"}
Oct 30 20:44:29.449373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.449310    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg" event={"ID":"3903830d-6c6a-4f62-9f1a-9ed15aea1d41","Type":"ContainerDied","Data":"c87773b5dcc07f512a7aa5606842b6cf19ad347e7de503197763d7281fcf1a8d"}
Oct 30 20:44:29.449373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.449333    5967 scope.go:122] "RemoveContainer" containerID="64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef"
Oct 30 20:44:29.461539 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.461523    5967 scope.go:122] "RemoveContainer" containerID="8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"
Oct 30 20:44:29.471199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.471180    5967 scope.go:122] "RemoveContainer" containerID="64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef"
Oct 30 20:44:29.471495 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:44:29.471474    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef\": container with ID starting with 64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef not found: ID does not exist" containerID="64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef"
Oct 30 20:44:29.471560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.471505    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef"} err="failed to get container status \"64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef\": rpc error: code = NotFound desc = could not find container \"64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef\": container with ID starting with 64bd2fe235e5516a9da11c757210c1bbd771e28e055aac0393452f73ed030bef not found: ID does not exist"
Oct 30 20:44:29.471560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.471523    5967 scope.go:122] "RemoveContainer" containerID="8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"
Oct 30 20:44:29.471803 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:44:29.471756    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad\": container with ID starting with 8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad not found: ID does not exist" containerID="8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"
Oct 30 20:44:29.471803 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.471775    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad"} err="failed to get container status \"8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad\": rpc error: code = NotFound desc = could not find container \"8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad\": container with ID starting with 8728e095a2c259c1ba66c268193851142d282660a6ba606cf525dd4ac7c957ad not found: ID does not exist"
Oct 30 20:44:29.616190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.616162    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"]
Oct 30 20:44:29.666365 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:29.666340    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg"]
Oct 30 20:44:30.149964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:30.149932    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"]
Oct 30 20:44:30.156852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:30.156818    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23cde229-vdxms"]
Oct 30 20:44:30.227776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:30.227719    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"]
Oct 30 20:44:30.228063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:30.228034    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy" containerID="cri-o://32392bab6706f2cd69567a6abca13383104b0c984746fb4b11a153957ff5112f" gracePeriod=30
Oct 30 20:44:31.544524 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:31.544480    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" path="/var/lib/kubelet/pods/3903830d-6c6a-4f62-9f1a-9ed15aea1d41/volumes"
Oct 30 20:44:31.545210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:31.545188    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8649efd8-d749-4b51-bcc7-2356ff62e156" path="/var/lib/kubelet/pods/8649efd8-d749-4b51-bcc7-2356ff62e156/volumes"
Oct 30 20:44:31.982962 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:44:31.982926    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:44:35.626339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.626256    5967 generic.go:356] "Generic (PLEG): container finished" podID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerID="32392bab6706f2cd69567a6abca13383104b0c984746fb4b11a153957ff5112f" exitCode=0
Oct 30 20:44:35.627496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.626577    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" event={"ID":"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b","Type":"ContainerDied","Data":"32392bab6706f2cd69567a6abca13383104b0c984746fb4b11a153957ff5112f"}
Oct 30 20:44:35.627496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.626633    5967 scope.go:122] "RemoveContainer" containerID="8e08bdb41706159fbf1e29d725bc613db7c4f42c90b71306c72500f2e99e05e3"
Oct 30 20:44:35.674280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.674258    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:44:35.768988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.768963    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.769128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769039    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.769128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769096    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.769251 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769226    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:35.769289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769253    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.769384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769369    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:35.769684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769666    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.769684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.769683    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.771226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.771198    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:35.771542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.771301    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "kube-api-access-twhhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:35.870189 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870159    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.870355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870267    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.870355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870287    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.870467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870436    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.870519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870461    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:35.870519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870510    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs\") pod \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\" (UID: \"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b\") "
Oct 30 20:44:35.870979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870943    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:35.870979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.870969    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:35.871401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.871247    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.871401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.871270    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.871401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.871285    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhhq\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-kube-api-access-twhhq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.871401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.871353    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.872121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.871828    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.873105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.873085    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:35.873234 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.873118    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo" pod "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" (UID: "5ef7d8ed-4ba6-4117-a0f7-76165a09e72b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:44:35.973025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.972989    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.973025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:35.973020    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:36.635504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:36.635434    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"
Oct 30 20:44:36.635504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:36.635480    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs" event={"ID":"5ef7d8ed-4ba6-4117-a0f7-76165a09e72b","Type":"ContainerDied","Data":"96f0d83d6286bb09a6fbef5fccd61ca0194250de17bf9932988c6b1ff722370f"}
Oct 30 20:44:36.636006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:36.635526    5967 scope.go:122] "RemoveContainer" containerID="32392bab6706f2cd69567a6abca13383104b0c984746fb4b11a153957ff5112f"
Oct 30 20:44:36.916401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:36.916327    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"]
Oct 30 20:44:36.922482 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:36.922450    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs"]
Oct 30 20:44:37.544385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:37.544349    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" path="/var/lib/kubelet/pods/5ef7d8ed-4ba6-4117-a0f7-76165a09e72b/volumes"
Oct 30 20:44:42.275502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.275472    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"]
Oct 30 20:44:42.276223 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276206    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" containerName="payload-processing"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276225    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" containerName="payload-processing"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276239    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276244    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276254    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f29a78b7-1351-4fe0-bcca-29ed0a717347" containerName="payload-pre-processing"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276259    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29a78b7-1351-4fe0-bcca-29ed0a717347" containerName="payload-pre-processing"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276268    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ddd13fe6-31fc-4690-92cc-bea3ad21430a" containerName="payload-pre-processing"
Oct 30 20:44:42.276272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276272    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd13fe6-31fc-4690-92cc-bea3ad21430a" containerName="payload-pre-processing"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276278    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8649efd8-d749-4b51-bcc7-2356ff62e156" containerName="revoke-keys"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276282    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8649efd8-d749-4b51-bcc7-2356ff62e156" containerName="revoke-keys"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276292    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276296    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276306    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276310    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276317    5967 cpu_manager.go:414] "containerMap: removing container" podUID="044de5a6-9abd-4ea5-89ff-720297490928" containerName="revoke-keys"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276321    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044de5a6-9abd-4ea5-89ff-720297490928" containerName="revoke-keys"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276334    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276338    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276453    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b6cddc-f1f5-41a1-8775-a74b0386bfdd" containerName="payload-processing"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276463    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276469    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29a78b7-1351-4fe0-bcca-29ed0a717347" containerName="payload-pre-processing"
Oct 30 20:44:42.276471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276474    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd13fe6-31fc-4690-92cc-bea3ad21430a" containerName="payload-pre-processing"
Oct 30 20:44:42.276910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276481    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8649efd8-d749-4b51-bcc7-2356ff62e156" containerName="revoke-keys"
Oct 30 20:44:42.276910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276487    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy"
Oct 30 20:44:42.276910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276495    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef7d8ed-4ba6-4117-a0f7-76165a09e72b" containerName="istio-proxy"
Oct 30 20:44:42.276910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276500    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="044de5a6-9abd-4ea5-89ff-720297490928" containerName="revoke-keys"
Oct 30 20:44:42.276910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.276508    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3903830d-6c6a-4f62-9f1a-9ed15aea1d41" containerName="istio-proxy"
Oct 30 20:44:42.280082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.280065    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.343435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.343403    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"]
Oct 30 20:44:42.357244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.357212    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjgz\" (UniqueName: \"kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.357492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.357472    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.357606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.357588    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.386972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.386941    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:44:42.386972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.386946    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:44:42.386972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.386958    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:44:42.387206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.386987    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:44:42.459408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.459375    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.459580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.459471    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.459623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.459601    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjgz\" (UniqueName: \"kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.460142 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.460118    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.461903 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.461881    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.518583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.518547    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjgz\" (UniqueName: \"kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz\") pod \"maas-api-revoke-sub-806d51b0-ztvsg\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.593987 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.593918    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:42.975718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:42.975691    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"]
Oct 30 20:44:42.976019 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:44:42.975998    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4fdbbc_f573_41a6_8b16_91213479784f.slice/crio-d6573fd14ec40a3129aaafd2424554a71525e5c8f790e466d67cfe35aab0bafc WatchSource:0}: Error finding container d6573fd14ec40a3129aaafd2424554a71525e5c8f790e466d67cfe35aab0bafc: Status 404 returned error can't find the container with id d6573fd14ec40a3129aaafd2424554a71525e5c8f790e466d67cfe35aab0bafc
Oct 30 20:44:43.702493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:43.702399    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg" event={"ID":"6f4fdbbc-f573-41a6-8b16-91213479784f","Type":"ContainerStarted","Data":"8adc90ef2f9f2715b8ac180a2d7d166774299e484ebfa937793a1edf6c0163af"}
Oct 30 20:44:43.702493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:43.702461    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg" event={"ID":"6f4fdbbc-f573-41a6-8b16-91213479784f","Type":"ContainerStarted","Data":"d6573fd14ec40a3129aaafd2424554a71525e5c8f790e466d67cfe35aab0bafc"}
Oct 30 20:44:43.820475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:43.820416    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg" podStartSLOduration=1.820400508 podStartE2EDuration="1.820400508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:43.815899283 +0000 UTC m=+1464.416701242" watchObservedRunningTime="2026-10-30 20:44:43.820400508 +0000 UTC m=+1464.421202467"
Oct 30 20:44:44.722923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:44.722890    5967 generic.go:356] "Generic (PLEG): container finished" podID="6f4fdbbc-f573-41a6-8b16-91213479784f" containerID="8adc90ef2f9f2715b8ac180a2d7d166774299e484ebfa937793a1edf6c0163af" exitCode=0
Oct 30 20:44:44.723403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:44.723163    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg" event={"ID":"6f4fdbbc-f573-41a6-8b16-91213479784f","Type":"ContainerDied","Data":"8adc90ef2f9f2715b8ac180a2d7d166774299e484ebfa937793a1edf6c0163af"}
Oct 30 20:44:45.903148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:45.903123    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:46.008981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.008897    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca\") pod \"6f4fdbbc-f573-41a6-8b16-91213479784f\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") "
Oct 30 20:44:46.008981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.008969    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz\" (UniqueName: \"kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz\") pod \"6f4fdbbc-f573-41a6-8b16-91213479784f\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") "
Oct 30 20:44:46.009165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.009107    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca\") pod \"6f4fdbbc-f573-41a6-8b16-91213479784f\" (UID: \"6f4fdbbc-f573-41a6-8b16-91213479784f\") "
Oct 30 20:44:46.009326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.009302    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca" pod "6f4fdbbc-f573-41a6-8b16-91213479784f" (UID: "6f4fdbbc-f573-41a6-8b16-91213479784f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:46.009621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.009595    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:46.011175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.011147    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca" pod "6f4fdbbc-f573-41a6-8b16-91213479784f" (UID: "6f4fdbbc-f573-41a6-8b16-91213479784f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:46.011270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.011202    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz" pod "6f4fdbbc-f573-41a6-8b16-91213479784f" (UID: "6f4fdbbc-f573-41a6-8b16-91213479784f"). InnerVolumeSpecName "kube-api-access-8bjgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:46.124770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.122975    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bjgz\" (UniqueName: \"kubernetes.io/projected/6f4fdbbc-f573-41a6-8b16-91213479784f-kube-api-access-8bjgz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:46.124770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.123017    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4fdbbc-f573-41a6-8b16-91213479784f-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:46.743974 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.743944    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"
Oct 30 20:44:46.744281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.743962    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg" event={"ID":"6f4fdbbc-f573-41a6-8b16-91213479784f","Type":"ContainerDied","Data":"d6573fd14ec40a3129aaafd2424554a71525e5c8f790e466d67cfe35aab0bafc"}
Oct 30 20:44:46.744386 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:46.744311    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6573fd14ec40a3129aaafd2424554a71525e5c8f790e466d67cfe35aab0bafc"
Oct 30 20:44:57.047665 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:57.047631    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"]
Oct 30 20:44:57.096316 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:57.096285    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966d45c1-z4l7j"]
Oct 30 20:44:57.544009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:44:57.543982    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422bc0c3-948f-4499-a095-0593e0582788" path="/var/lib/kubelet/pods/422bc0c3-948f-4499-a095-0593e0582788/volumes"
Oct 30 20:45:00.182758 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.182701    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"]
Oct 30 20:45:00.183881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.183856    5967 cpu_manager.go:414] "containerMap: removing container" podUID="6f4fdbbc-f573-41a6-8b16-91213479784f" containerName="revoke-keys"
Oct 30 20:45:00.183881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.183880    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4fdbbc-f573-41a6-8b16-91213479784f" containerName="revoke-keys"
Oct 30 20:45:00.184148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.184134    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4fdbbc-f573-41a6-8b16-91213479784f" containerName="revoke-keys"
Oct 30 20:45:00.189910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.189887    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"]
Oct 30 20:45:00.190024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.189983    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.195444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.195362    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"]
Oct 30 20:45:00.195772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.195719    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.200468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.200448    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.232337 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.232315    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"]
Oct 30 20:45:00.236483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.236467    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.237085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.237063    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"]
Oct 30 20:45:00.241255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.241238    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.242045 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.241914    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:45:00.242045 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.242024    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:45:00.242657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.242638    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"]
Oct 30 20:45:00.247907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.247506    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.277128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.277107    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:45:00.288890 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.287860    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"]
Oct 30 20:45:00.298958 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.298178    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"]
Oct 30 20:45:00.298958 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.298245    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"]
Oct 30 20:45:00.304759 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.304539    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"]
Oct 30 20:45:00.323271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323234    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.323500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323394    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdr2h\" (UniqueName: \"kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.323500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323433    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.323500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323467    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.323670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323509    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wdx\" (UniqueName: \"kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.323670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323656    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt9p7\" (UniqueName: \"kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7\") pod \"maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.323981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.323958    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.324048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.324028    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.324131 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.324114    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzn7\" (UniqueName: \"kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.324184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.324171    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlxx\" (UniqueName: \"kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.336538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.336504    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"]
Oct 30 20:45:00.353238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.353204    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"]
Oct 30 20:45:00.426095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426061    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wdx\" (UniqueName: \"kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.426292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426103    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9p7\" (UniqueName: \"kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7\") pod \"maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.426292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426150    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-8n9t7\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.426292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426275    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.426484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426329    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxmb\" (UniqueName: \"kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb\") pod \"maas-api-key-cleanup-29889885-8n9t7\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.426484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426368    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.426484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426445    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzn7\" (UniqueName: \"kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.426484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426478    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlxx\" (UniqueName: \"kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.426697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426674    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.426788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426699    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdr2h\" (UniqueName: \"kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.426788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426721    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.426788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.426771    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.427088 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.427066    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.427161 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.427092    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.427369 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.427352    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.427557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.427533    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.427639 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.427621    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.482870 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.482843    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wdx\" (UniqueName: \"kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.483020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.482846    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdr2h\" (UniqueName: \"kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.483266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.483240    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzn7\" (UniqueName: \"kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.483581 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.483554    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9p7\" (UniqueName: \"kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7\") pod \"maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.484557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.484529    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlxx\" (UniqueName: \"kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.508307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.508283    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:00.516512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.516476    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:00.523658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.523636    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:00.528693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.528669    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-8n9t7\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.528928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.528868    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxmb\" (UniqueName: \"kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb\") pod \"maas-api-key-cleanup-29889885-8n9t7\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.529770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.529728    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-8n9t7\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.557675 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.557647    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:00.565280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.565259    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:00.609262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.600656    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxmb\" (UniqueName: \"kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb\") pod \"maas-api-key-cleanup-29889885-8n9t7\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.872117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.872084    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:00.910652 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:00.910618    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd165444a_a39a_472d_9c93_520de9bda277.slice/crio-f0141b9c4fe83250b9c2f7dd3a76aaefb6d6ed7dc95e94fc1660d00ef97f7a9b WatchSource:0}: Error finding container f0141b9c4fe83250b9c2f7dd3a76aaefb6d6ed7dc95e94fc1660d00ef97f7a9b: Status 404 returned error can't find the container with id f0141b9c4fe83250b9c2f7dd3a76aaefb6d6ed7dc95e94fc1660d00ef97f7a9b
Oct 30 20:45:00.916979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.916947    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"]
Oct 30 20:45:00.972857 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.972101    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"]
Oct 30 20:45:00.989372 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:00.989305    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07558179_69bc_45c3_b4cb_8191d8174c94.slice/crio-2fc95137666d1f9efdc4c9ce243a4aa4bf6f1be19f1a2d7cf869d7edda45b71e WatchSource:0}: Error finding container 2fc95137666d1f9efdc4c9ce243a4aa4bf6f1be19f1a2d7cf869d7edda45b71e: Status 404 returned error can't find the container with id 2fc95137666d1f9efdc4c9ce243a4aa4bf6f1be19f1a2d7cf869d7edda45b71e
Oct 30 20:45:00.998177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:00.997215    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"]
Oct 30 20:45:01.072490 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:01.072461    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-1503b796a38d231b26f721a0e8249aed4ba3e33212c755e231dc9072dd75bc7c WatchSource:0}: Error finding container 1503b796a38d231b26f721a0e8249aed4ba3e33212c755e231dc9072dd75bc7c: Status 404 returned error can't find the container with id 1503b796a38d231b26f721a0e8249aed4ba3e33212c755e231dc9072dd75bc7c
Oct 30 20:45:01.073314 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:01.073288    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b56850_03be_492d_a5fe_22dd65cdb8ee.slice/crio-99c257f238becdee204ccce7970b98fd128a982409a2f0b4714397ddb38133a0 WatchSource:0}: Error finding container 99c257f238becdee204ccce7970b98fd128a982409a2f0b4714397ddb38133a0: Status 404 returned error can't find the container with id 99c257f238becdee204ccce7970b98fd128a982409a2f0b4714397ddb38133a0
Oct 30 20:45:01.074586 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.074564    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"]
Oct 30 20:45:01.080042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.080023    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"]
Oct 30 20:45:01.286682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.286658    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"]
Oct 30 20:45:01.373292 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:01.373249    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c734c66_ccde_4387_bb02_df9d805e817c.slice/crio-c7db70f0e94fb0aee4dd4529f2a6dd64b35b948bba6e52ec949ec9b895b842d7 WatchSource:0}: Error finding container c7db70f0e94fb0aee4dd4529f2a6dd64b35b948bba6e52ec949ec9b895b842d7: Status 404 returned error can't find the container with id c7db70f0e94fb0aee4dd4529f2a6dd64b35b948bba6e52ec949ec9b895b842d7
Oct 30 20:45:01.879110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.879030    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerStarted","Data":"4ff3b7ad3b10138aafdd7963a204c7feb8026ab93575d3aa8313817e5add6ba6"}
Oct 30 20:45:01.879110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.879100    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerStarted","Data":"1503b796a38d231b26f721a0e8249aed4ba3e33212c755e231dc9072dd75bc7c"}
Oct 30 20:45:01.880979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.880948    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerStarted","Data":"ee3578ebbdae10ad0696c1529a7ea97f63941107a86c30344868b7333da0e403"}
Oct 30 20:45:01.881101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.881005    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerStarted","Data":"99c257f238becdee204ccce7970b98fd128a982409a2f0b4714397ddb38133a0"}
Oct 30 20:45:01.883109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.883087    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerStarted","Data":"81732ea932af7335382bbc7200b3d371664196ccfb0dd3196b8d1e0d91564a00"}
Oct 30 20:45:01.883223 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.883132    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerStarted","Data":"f0141b9c4fe83250b9c2f7dd3a76aaefb6d6ed7dc95e94fc1660d00ef97f7a9b"}
Oct 30 20:45:01.884756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.884720    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerStarted","Data":"85fb98e3ac573524e68d13ccedfbdf0579052b748f0bb676a879999992a480f5"}
Oct 30 20:45:01.885390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.884781    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerStarted","Data":"c7db70f0e94fb0aee4dd4529f2a6dd64b35b948bba6e52ec949ec9b895b842d7"}
Oct 30 20:45:01.886406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.886384    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerStarted","Data":"a091acabcc2c64a55dadaa2ddd90d6582c6eb327bface7986968d721d7f02e4e"}
Oct 30 20:45:01.886521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.886427    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerStarted","Data":"2fc95137666d1f9efdc4c9ce243a4aa4bf6f1be19f1a2d7cf869d7edda45b71e"}
Oct 30 20:45:01.887832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.887812    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerStarted","Data":"6a9f80ff85415af40e23ebd68059b6f0dfbeee730b3f49ca47a84c1cf368ae4f"}
Oct 30 20:45:01.887926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.887853    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerStarted","Data":"58260eb77f230fdfc4caf9fdeef56b021284dfedb4b7a71c7c298d1b96eccb35"}
Oct 30 20:45:01.984977 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:01.984946    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:45:01.996056 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:01.996009    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" podStartSLOduration=1.9959947850000002 podStartE2EDuration="1.995994785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:01.992295124 +0000 UTC m=+1482.593097083" watchObservedRunningTime="2026-10-30 20:45:01.995994785 +0000 UTC m=+1482.596796787"
Oct 30 20:45:02.109991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.109946    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" podStartSLOduration=2.109932453 podStartE2EDuration="2.109932453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.105611884 +0000 UTC m=+1482.706413845" watchObservedRunningTime="2026-10-30 20:45:02.109932453 +0000 UTC m=+1482.710734413"
Oct 30 20:45:02.220920 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.220879    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" podStartSLOduration=2.220865973 podStartE2EDuration="2.220865973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.218044149 +0000 UTC m=+1482.818846109" watchObservedRunningTime="2026-10-30 20:45:02.220865973 +0000 UTC m=+1482.821667941"
Oct 30 20:45:02.333002 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.332959    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" podStartSLOduration=2.332945678 podStartE2EDuration="2.332945678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.331478096 +0000 UTC m=+1482.932280059" watchObservedRunningTime="2026-10-30 20:45:02.332945678 +0000 UTC m=+1482.933747637"
Oct 30 20:45:02.446871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.446776    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" podStartSLOduration=2.446730165 podStartE2EDuration="2.446730165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.443256149 +0000 UTC m=+1483.044058110" watchObservedRunningTime="2026-10-30 20:45:02.446730165 +0000 UTC m=+1483.047532125"
Oct 30 20:45:02.559841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.559789    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" podStartSLOduration=2.559776459 podStartE2EDuration="2.559776459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.556147744 +0000 UTC m=+1483.156949705" watchObservedRunningTime="2026-10-30 20:45:02.559776459 +0000 UTC m=+1483.160578418"
Oct 30 20:45:02.900088 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.900047    5967 generic.go:356] "Generic (PLEG): container finished" podID="7c734c66-ccde-4387-bb02-df9d805e817c" containerID="85fb98e3ac573524e68d13ccedfbdf0579052b748f0bb676a879999992a480f5" exitCode=60
Oct 30 20:45:02.900280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.900175    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerDied","Data":"85fb98e3ac573524e68d13ccedfbdf0579052b748f0bb676a879999992a480f5"}
Oct 30 20:45:02.900434 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.900389    5967 scope.go:122] "RemoveContainer" containerID="85fb98e3ac573524e68d13ccedfbdf0579052b748f0bb676a879999992a480f5"
Oct 30 20:45:02.901885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.901858    5967 generic.go:356] "Generic (PLEG): container finished" podID="07558179-69bc-45c3-b4cb-8191d8174c94" containerID="a091acabcc2c64a55dadaa2ddd90d6582c6eb327bface7986968d721d7f02e4e" exitCode=60
Oct 30 20:45:02.902017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.901894    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerDied","Data":"a091acabcc2c64a55dadaa2ddd90d6582c6eb327bface7986968d721d7f02e4e"}
Oct 30 20:45:02.902178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.902162    5967 scope.go:122] "RemoveContainer" containerID="a091acabcc2c64a55dadaa2ddd90d6582c6eb327bface7986968d721d7f02e4e"
Oct 30 20:45:02.908604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.908581    5967 generic.go:356] "Generic (PLEG): container finished" podID="f054465e-403b-4d08-aa4d-b148089359de" containerID="6a9f80ff85415af40e23ebd68059b6f0dfbeee730b3f49ca47a84c1cf368ae4f" exitCode=60
Oct 30 20:45:02.908781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.908674    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerDied","Data":"6a9f80ff85415af40e23ebd68059b6f0dfbeee730b3f49ca47a84c1cf368ae4f"}
Oct 30 20:45:02.909005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.908992    5967 scope.go:122] "RemoveContainer" containerID="6a9f80ff85415af40e23ebd68059b6f0dfbeee730b3f49ca47a84c1cf368ae4f"
Oct 30 20:45:02.911080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.911060    5967 generic.go:356] "Generic (PLEG): container finished" podID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerID="4ff3b7ad3b10138aafdd7963a204c7feb8026ab93575d3aa8313817e5add6ba6" exitCode=60
Oct 30 20:45:02.911170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.911142    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerDied","Data":"4ff3b7ad3b10138aafdd7963a204c7feb8026ab93575d3aa8313817e5add6ba6"}
Oct 30 20:45:02.911363 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.911348    5967 scope.go:122] "RemoveContainer" containerID="4ff3b7ad3b10138aafdd7963a204c7feb8026ab93575d3aa8313817e5add6ba6"
Oct 30 20:45:02.912976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.912955    5967 generic.go:356] "Generic (PLEG): container finished" podID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerID="ee3578ebbdae10ad0696c1529a7ea97f63941107a86c30344868b7333da0e403" exitCode=60
Oct 30 20:45:02.913068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.913021    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerDied","Data":"ee3578ebbdae10ad0696c1529a7ea97f63941107a86c30344868b7333da0e403"}
Oct 30 20:45:02.913239 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.913223    5967 scope.go:122] "RemoveContainer" containerID="ee3578ebbdae10ad0696c1529a7ea97f63941107a86c30344868b7333da0e403"
Oct 30 20:45:02.915063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.915043    5967 generic.go:356] "Generic (PLEG): container finished" podID="d165444a-a39a-472d-9c93-520de9bda277" containerID="81732ea932af7335382bbc7200b3d371664196ccfb0dd3196b8d1e0d91564a00" exitCode=60
Oct 30 20:45:02.915140 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.915122    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerDied","Data":"81732ea932af7335382bbc7200b3d371664196ccfb0dd3196b8d1e0d91564a00"}
Oct 30 20:45:02.915287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:02.915276    5967 scope.go:122] "RemoveContainer" containerID="81732ea932af7335382bbc7200b3d371664196ccfb0dd3196b8d1e0d91564a00"
Oct 30 20:45:03.926059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.926020    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerStarted","Data":"b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a"}
Oct 30 20:45:03.928177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.928149    5967 generic.go:356] "Generic (PLEG): container finished" podID="f054465e-403b-4d08-aa4d-b148089359de" containerID="a4e6f94e6b51115307a5b4736266db53e1698e2c11a8e19cefa9a0fd5c20df20" exitCode=60
Oct 30 20:45:03.928288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.928221    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerDied","Data":"a4e6f94e6b51115307a5b4736266db53e1698e2c11a8e19cefa9a0fd5c20df20"}
Oct 30 20:45:03.928288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.928277    5967 scope.go:122] "RemoveContainer" containerID="6a9f80ff85415af40e23ebd68059b6f0dfbeee730b3f49ca47a84c1cf368ae4f"
Oct 30 20:45:03.928576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.928557    5967 scope.go:122] "RemoveContainer" containerID="a4e6f94e6b51115307a5b4736266db53e1698e2c11a8e19cefa9a0fd5c20df20"
Oct 30 20:45:03.928998 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:03.928970    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm_odh-ai-gateway-infra(f054465e-403b-4d08-aa4d-b148089359de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" podUID="f054465e-403b-4d08-aa4d-b148089359de"
Oct 30 20:45:03.930199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.930179    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerStarted","Data":"64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b"}
Oct 30 20:45:03.932174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.932150    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerStarted","Data":"149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b"}
Oct 30 20:45:03.934241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.934220    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerStarted","Data":"f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a"}
Oct 30 20:45:03.936128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:03.936110    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerStarted","Data":"eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3"}
Oct 30 20:45:04.946546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.946514    5967 generic.go:356] "Generic (PLEG): container finished" podID="07558179-69bc-45c3-b4cb-8191d8174c94" containerID="b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a" exitCode=60
Oct 30 20:45:04.947038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.946600    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerDied","Data":"b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a"}
Oct 30 20:45:04.947038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.946665    5967 scope.go:122] "RemoveContainer" containerID="a091acabcc2c64a55dadaa2ddd90d6582c6eb327bface7986968d721d7f02e4e"
Oct 30 20:45:04.947038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.946873    5967 scope.go:122] "RemoveContainer" containerID="b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a"
Oct 30 20:45:04.947219 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:04.947178    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c_odh-ai-gateway-infra(07558179-69bc-45c3-b4cb-8191d8174c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" podUID="07558179-69bc-45c3-b4cb-8191d8174c94"
Oct 30 20:45:04.948733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.948715    5967 scope.go:122] "RemoveContainer" containerID="a4e6f94e6b51115307a5b4736266db53e1698e2c11a8e19cefa9a0fd5c20df20"
Oct 30 20:45:04.949050 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:04.949030    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm_odh-ai-gateway-infra(f054465e-403b-4d08-aa4d-b148089359de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" podUID="f054465e-403b-4d08-aa4d-b148089359de"
Oct 30 20:45:04.950663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.950642    5967 generic.go:356] "Generic (PLEG): container finished" podID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerID="64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b" exitCode=60
Oct 30 20:45:04.950792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.950716    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerDied","Data":"64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b"}
Oct 30 20:45:04.950992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.950977    5967 scope.go:122] "RemoveContainer" containerID="64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b"
Oct 30 20:45:04.951221 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:04.951197    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp_odh-ai-gateway-infra(7520efd8-ede8-4603-9b17-1f10fbd03119)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119"
Oct 30 20:45:04.952956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.952935    5967 generic.go:356] "Generic (PLEG): container finished" podID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerID="149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b" exitCode=60
Oct 30 20:45:04.953073 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.953020    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerDied","Data":"149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b"}
Oct 30 20:45:04.953189 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.953169    5967 scope.go:122] "RemoveContainer" containerID="149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b"
Oct 30 20:45:04.953389 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:04.953371    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc_odh-ai-gateway-infra(83b56850-03be-492d-a5fe-22dd65cdb8ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee"
Oct 30 20:45:04.955154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.955131    5967 generic.go:356] "Generic (PLEG): container finished" podID="d165444a-a39a-472d-9c93-520de9bda277" containerID="f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a" exitCode=60
Oct 30 20:45:04.955305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.955276    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerDied","Data":"f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a"}
Oct 30 20:45:04.955442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.955387    5967 scope.go:122] "RemoveContainer" containerID="f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a"
Oct 30 20:45:04.955605 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:04.955579    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l_odh-ai-gateway-infra(d165444a-a39a-472d-9c93-520de9bda277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" podUID="d165444a-a39a-472d-9c93-520de9bda277"
Oct 30 20:45:04.957451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.957428    5967 generic.go:356] "Generic (PLEG): container finished" podID="7c734c66-ccde-4387-bb02-df9d805e817c" containerID="eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3" exitCode=60
Oct 30 20:45:04.957550 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.957481    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerDied","Data":"eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3"}
Oct 30 20:45:04.957657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.957645    5967 scope.go:122] "RemoveContainer" containerID="eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3"
Oct 30 20:45:04.957868 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:04.957851    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-8n9t7_odh-ai-gateway-infra(7c734c66-ccde-4387-bb02-df9d805e817c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" podUID="7c734c66-ccde-4387-bb02-df9d805e817c"
Oct 30 20:45:04.962061 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.962047    5967 scope.go:122] "RemoveContainer" containerID="4ff3b7ad3b10138aafdd7963a204c7feb8026ab93575d3aa8313817e5add6ba6"
Oct 30 20:45:04.976555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.976538    5967 scope.go:122] "RemoveContainer" containerID="ee3578ebbdae10ad0696c1529a7ea97f63941107a86c30344868b7333da0e403"
Oct 30 20:45:04.989823 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:04.989803    5967 scope.go:122] "RemoveContainer" containerID="81732ea932af7335382bbc7200b3d371664196ccfb0dd3196b8d1e0d91564a00"
Oct 30 20:45:05.003424 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:05.003408    5967 scope.go:122] "RemoveContainer" containerID="85fb98e3ac573524e68d13ccedfbdf0579052b748f0bb676a879999992a480f5"
Oct 30 20:45:05.967871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:05.967825    5967 scope.go:122] "RemoveContainer" containerID="149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b"
Oct 30 20:45:05.968346 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:05.968162    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc_odh-ai-gateway-infra(83b56850-03be-492d-a5fe-22dd65cdb8ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee"
Oct 30 20:45:05.969634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:05.969613    5967 scope.go:122] "RemoveContainer" containerID="f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a"
Oct 30 20:45:05.969919 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:05.969896    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l_odh-ai-gateway-infra(d165444a-a39a-472d-9c93-520de9bda277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" podUID="d165444a-a39a-472d-9c93-520de9bda277"
Oct 30 20:45:05.971229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:05.971210    5967 scope.go:122] "RemoveContainer" containerID="eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3"
Oct 30 20:45:05.971479 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:05.971459    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-8n9t7_odh-ai-gateway-infra(7c734c66-ccde-4387-bb02-df9d805e817c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" podUID="7c734c66-ccde-4387-bb02-df9d805e817c"
Oct 30 20:45:05.972729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:05.972707    5967 scope.go:122] "RemoveContainer" containerID="b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a"
Oct 30 20:45:05.973003 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:05.972980    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c_odh-ai-gateway-infra(07558179-69bc-45c3-b4cb-8191d8174c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" podUID="07558179-69bc-45c3-b4cb-8191d8174c94"
Oct 30 20:45:05.974264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:05.974243    5967 scope.go:122] "RemoveContainer" containerID="64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b"
Oct 30 20:45:05.974450 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:05.974434    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp_odh-ai-gateway-infra(7520efd8-ede8-4603-9b17-1f10fbd03119)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119"
Oct 30 20:45:08.050160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:08.050132    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"]
Oct 30 20:45:08.099550 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:08.099521    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b800f7b2-nsptn"]
Oct 30 20:45:09.543883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:09.543848    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2694404-ebea-48a2-b32a-67a2da8358fc" path="/var/lib/kubelet/pods/b2694404-ebea-48a2-b32a-67a2da8358fc/volumes"
Oct 30 20:45:13.055231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:13.055174    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"]
Oct 30 20:45:13.085340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:13.085310    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7efc2280-5bhv7"]
Oct 30 20:45:13.544134 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:13.544104    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02c7f07-0093-40cb-8d31-b760f404af6e" path="/var/lib/kubelet/pods/f02c7f07-0093-40cb-8d31-b760f404af6e/volumes"
Oct 30 20:45:16.536315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:16.536280    5967 scope.go:122] "RemoveContainer" containerID="b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a"
Oct 30 20:45:17.157787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:17.155105    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerStarted","Data":"65a75c9f19318c30ec402b394878edb1ac2a7f41ad21da5b35353591810e03f0"}
Oct 30 20:45:17.538931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:17.537842    5967 scope.go:122] "RemoveContainer" containerID="149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b"
Oct 30 20:45:17.540169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:17.540150    5967 scope.go:122] "RemoveContainer" containerID="eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3"
Oct 30 20:45:18.165696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.165665    5967 generic.go:356] "Generic (PLEG): container finished" podID="07558179-69bc-45c3-b4cb-8191d8174c94" containerID="65a75c9f19318c30ec402b394878edb1ac2a7f41ad21da5b35353591810e03f0" exitCode=60
Oct 30 20:45:18.165918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.165718    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerDied","Data":"65a75c9f19318c30ec402b394878edb1ac2a7f41ad21da5b35353591810e03f0"}
Oct 30 20:45:18.165918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.165783    5967 scope.go:122] "RemoveContainer" containerID="b48ab7f6af1a6d9604740ec063d03262975f373cc4c49b3f95cc2cc6313a0f4a"
Oct 30 20:45:18.166176 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.166153    5967 scope.go:122] "RemoveContainer" containerID="65a75c9f19318c30ec402b394878edb1ac2a7f41ad21da5b35353591810e03f0"
Oct 30 20:45:18.166564 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:18.166536    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c_odh-ai-gateway-infra(07558179-69bc-45c3-b4cb-8191d8174c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" podUID="07558179-69bc-45c3-b4cb-8191d8174c94"
Oct 30 20:45:18.295554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.295525    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"]
Oct 30 20:45:18.536176 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.536088    5967 scope.go:122] "RemoveContainer" containerID="f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a"
Oct 30 20:45:18.536630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.536605    5967 scope.go:122] "RemoveContainer" containerID="a4e6f94e6b51115307a5b4736266db53e1698e2c11a8e19cefa9a0fd5c20df20"
Oct 30 20:45:18.537635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:18.537610    5967 scope.go:122] "RemoveContainer" containerID="64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b"
Oct 30 20:45:19.181205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.181170    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerStarted","Data":"0cf22b3b9c7abce969253d31c10d0df7e998ddf70b7add283e35a8adb20b7bbf"}
Oct 30 20:45:19.185574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.185544    5967 generic.go:356] "Generic (PLEG): container finished" podID="7c734c66-ccde-4387-bb02-df9d805e817c" containerID="6103960f4ea61fee6812f4b6d2d4a51f7b08920e6ad3bd0003ffb0640731d93e" exitCode=60
Oct 30 20:45:19.185699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.185621    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerDied","Data":"6103960f4ea61fee6812f4b6d2d4a51f7b08920e6ad3bd0003ffb0640731d93e"}
Oct 30 20:45:19.185815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.185700    5967 scope.go:122] "RemoveContainer" containerID="eeb46d4e0d10c75dcd7ff7fed0ef74092c5d44166758754fc6c59e02354f4de3"
Oct 30 20:45:19.187001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.186977    5967 scope.go:122] "RemoveContainer" containerID="6103960f4ea61fee6812f4b6d2d4a51f7b08920e6ad3bd0003ffb0640731d93e"
Oct 30 20:45:19.187732 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:19.187704    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-8n9t7_odh-ai-gateway-infra(7c734c66-ccde-4387-bb02-df9d805e817c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" podUID="7c734c66-ccde-4387-bb02-df9d805e817c"
Oct 30 20:45:19.192728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.192701    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerStarted","Data":"d7b488cff5375710d4574b7af7bfc4acdbf956ddd28048c0e33c9cd91995bbf8"}
Oct 30 20:45:19.195117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.195090    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerStarted","Data":"02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4"}
Oct 30 20:45:19.197264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.197237    5967 generic.go:356] "Generic (PLEG): container finished" podID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerID="fa0f733814c51ed384a4dc43a83bf78b4bd87da88dfdcab590b3cd78d965f50f" exitCode=60
Oct 30 20:45:19.197346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.197300    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerDied","Data":"fa0f733814c51ed384a4dc43a83bf78b4bd87da88dfdcab590b3cd78d965f50f"}
Oct 30 20:45:19.197586 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.197575    5967 scope.go:122] "RemoveContainer" containerID="fa0f733814c51ed384a4dc43a83bf78b4bd87da88dfdcab590b3cd78d965f50f"
Oct 30 20:45:19.197899 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:19.197881    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc_odh-ai-gateway-infra(83b56850-03be-492d-a5fe-22dd65cdb8ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee"
Oct 30 20:45:19.202983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.202967    5967 scope.go:122] "RemoveContainer" containerID="149a9edfccf6e93b0691ae3f6b139b45afa4caa742dfef81236a470ab2e7454b"
Oct 30 20:45:19.352827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.352801    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:19.370017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.369985    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca\") pod \"07558179-69bc-45c3-b4cb-8191d8174c94\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") "
Oct 30 20:45:19.370295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.370273    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx\" (UniqueName: \"kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx\") pod \"07558179-69bc-45c3-b4cb-8191d8174c94\" (UID: \"07558179-69bc-45c3-b4cb-8191d8174c94\") "
Oct 30 20:45:19.370467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.370281    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca" pod "07558179-69bc-45c3-b4cb-8191d8174c94" (UID: "07558179-69bc-45c3-b4cb-8191d8174c94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:19.371100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.371057    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07558179-69bc-45c3-b4cb-8191d8174c94-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:19.373184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.373143    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx" pod "07558179-69bc-45c3-b4cb-8191d8174c94" (UID: "07558179-69bc-45c3-b4cb-8191d8174c94"). InnerVolumeSpecName "kube-api-access-qjlxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:19.472133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:19.472103    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlxx\" (UniqueName: \"kubernetes.io/projected/07558179-69bc-45c3-b4cb-8191d8174c94-kube-api-access-qjlxx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:20.207691 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.207657    5967 generic.go:356] "Generic (PLEG): container finished" podID="d165444a-a39a-472d-9c93-520de9bda277" containerID="0cf22b3b9c7abce969253d31c10d0df7e998ddf70b7add283e35a8adb20b7bbf" exitCode=60
Oct 30 20:45:20.208179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.207728    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerDied","Data":"0cf22b3b9c7abce969253d31c10d0df7e998ddf70b7add283e35a8adb20b7bbf"}
Oct 30 20:45:20.208179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.207815    5967 scope.go:122] "RemoveContainer" containerID="f8d3b1876b5ffb840459becca7c71d79b2cfc548a34c1f168f8003cdc5e0171a"
Oct 30 20:45:20.208179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.208147    5967 scope.go:122] "RemoveContainer" containerID="0cf22b3b9c7abce969253d31c10d0df7e998ddf70b7add283e35a8adb20b7bbf"
Oct 30 20:45:20.208416 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:20.208394    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l_odh-ai-gateway-infra(d165444a-a39a-472d-9c93-520de9bda277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" podUID="d165444a-a39a-472d-9c93-520de9bda277"
Oct 30 20:45:20.211403 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.211383    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"
Oct 30 20:45:20.211525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.211402    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c" event={"ID":"07558179-69bc-45c3-b4cb-8191d8174c94","Type":"ContainerDied","Data":"2fc95137666d1f9efdc4c9ce243a4aa4bf6f1be19f1a2d7cf869d7edda45b71e"}
Oct 30 20:45:20.213353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.213327    5967 generic.go:356] "Generic (PLEG): container finished" podID="f054465e-403b-4d08-aa4d-b148089359de" containerID="d7b488cff5375710d4574b7af7bfc4acdbf956ddd28048c0e33c9cd91995bbf8" exitCode=60
Oct 30 20:45:20.213446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.213351    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerDied","Data":"d7b488cff5375710d4574b7af7bfc4acdbf956ddd28048c0e33c9cd91995bbf8"}
Oct 30 20:45:20.213705 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.213686    5967 scope.go:122] "RemoveContainer" containerID="d7b488cff5375710d4574b7af7bfc4acdbf956ddd28048c0e33c9cd91995bbf8"
Oct 30 20:45:20.214287 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:20.213993    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm_odh-ai-gateway-infra(f054465e-403b-4d08-aa4d-b148089359de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" podUID="f054465e-403b-4d08-aa4d-b148089359de"
Oct 30 20:45:20.215449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.215430    5967 generic.go:356] "Generic (PLEG): container finished" podID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerID="02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4" exitCode=60
Oct 30 20:45:20.215517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.215498    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerDied","Data":"02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4"}
Oct 30 20:45:20.215835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.215817    5967 scope.go:122] "RemoveContainer" containerID="02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4"
Oct 30 20:45:20.216057 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:20.216038    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp_odh-ai-gateway-infra(7520efd8-ede8-4603-9b17-1f10fbd03119)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119"
Oct 30 20:45:20.227566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.227545    5967 scope.go:122] "RemoveContainer" containerID="65a75c9f19318c30ec402b394878edb1ac2a7f41ad21da5b35353591810e03f0"
Oct 30 20:45:20.237220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.237201    5967 scope.go:122] "RemoveContainer" containerID="a4e6f94e6b51115307a5b4736266db53e1698e2c11a8e19cefa9a0fd5c20df20"
Oct 30 20:45:20.252094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.252074    5967 scope.go:122] "RemoveContainer" containerID="64860e0d91490a824f65429c5c23d88585a690e23640e1268fc5552bfb2b6a0b"
Oct 30 20:45:20.318339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.318314    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"]
Oct 30 20:45:20.433667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.433637    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"]
Oct 30 20:45:20.552543 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.552508    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"]
Oct 30 20:45:20.585157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.585124    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"]
Oct 30 20:45:20.598718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.598693    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f9b95-29889885-62q2c"]
Oct 30 20:45:20.616868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.616845    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:20.662651 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.662627    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"]
Oct 30 20:45:20.687618 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.687594    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca\") pod \"83b56850-03be-492d-a5fe-22dd65cdb8ee\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") "
Oct 30 20:45:20.687778 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.687687    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7\" (UniqueName: \"kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7\") pod \"83b56850-03be-492d-a5fe-22dd65cdb8ee\" (UID: \"83b56850-03be-492d-a5fe-22dd65cdb8ee\") "
Oct 30 20:45:20.687975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.687954    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca" pod "83b56850-03be-492d-a5fe-22dd65cdb8ee" (UID: "83b56850-03be-492d-a5fe-22dd65cdb8ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:20.688310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.688296    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b56850-03be-492d-a5fe-22dd65cdb8ee-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:20.689945 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.689921    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7" pod "83b56850-03be-492d-a5fe-22dd65cdb8ee" (UID: "83b56850-03be-492d-a5fe-22dd65cdb8ee"). InnerVolumeSpecName "kube-api-access-jmzn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:20.790022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.789939    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzn7\" (UniqueName: \"kubernetes.io/projected/83b56850-03be-492d-a5fe-22dd65cdb8ee-kube-api-access-jmzn7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:20.844255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.844232    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:20.888846 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.888820    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"]
Oct 30 20:45:20.890514 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.890486    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca\") pod \"7c734c66-ccde-4387-bb02-df9d805e817c\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") "
Oct 30 20:45:20.890651 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.890553    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb\" (UniqueName: \"kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb\") pod \"7c734c66-ccde-4387-bb02-df9d805e817c\" (UID: \"7c734c66-ccde-4387-bb02-df9d805e817c\") "
Oct 30 20:45:20.891493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.891469    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca" pod "7c734c66-ccde-4387-bb02-df9d805e817c" (UID: "7c734c66-ccde-4387-bb02-df9d805e817c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:20.893374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.893347    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb" pod "7c734c66-ccde-4387-bb02-df9d805e817c" (UID: "7c734c66-ccde-4387-bb02-df9d805e817c"). InnerVolumeSpecName "kube-api-access-5dxmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:20.992374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.992346    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c734c66-ccde-4387-bb02-df9d805e817c-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:20.992374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:20.992369    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxmb\" (UniqueName: \"kubernetes.io/projected/7c734c66-ccde-4387-bb02-df9d805e817c-kube-api-access-5dxmb\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.229782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.229757    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"
Oct 30 20:45:21.230145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.229775    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7" event={"ID":"7c734c66-ccde-4387-bb02-df9d805e817c","Type":"ContainerDied","Data":"c7db70f0e94fb0aee4dd4529f2a6dd64b35b948bba6e52ec949ec9b895b842d7"}
Oct 30 20:45:21.230145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.229845    5967 scope.go:122] "RemoveContainer" containerID="6103960f4ea61fee6812f4b6d2d4a51f7b08920e6ad3bd0003ffb0640731d93e"
Oct 30 20:45:21.235613 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.235587    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"
Oct 30 20:45:21.235728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.235633    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc" event={"ID":"83b56850-03be-492d-a5fe-22dd65cdb8ee","Type":"ContainerDied","Data":"99c257f238becdee204ccce7970b98fd128a982409a2f0b4714397ddb38133a0"}
Oct 30 20:45:21.245853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.245835    5967 scope.go:122] "RemoveContainer" containerID="fa0f733814c51ed384a4dc43a83bf78b4bd87da88dfdcab590b3cd78d965f50f"
Oct 30 20:45:21.426684 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.426643    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:21.463808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.463785    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:21.473006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.472988    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:21.501762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.501685    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca\") pod \"d165444a-a39a-472d-9c93-520de9bda277\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") "
Oct 30 20:45:21.501843 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.501803    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h\" (UniqueName: \"kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h\") pod \"f054465e-403b-4d08-aa4d-b148089359de\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") "
Oct 30 20:45:21.501843 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.501834    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx\" (UniqueName: \"kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx\") pod \"d165444a-a39a-472d-9c93-520de9bda277\" (UID: \"d165444a-a39a-472d-9c93-520de9bda277\") "
Oct 30 20:45:21.501961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.501876    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca\") pod \"7520efd8-ede8-4603-9b17-1f10fbd03119\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") "
Oct 30 20:45:21.502013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502001    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca\") pod \"f054465e-403b-4d08-aa4d-b148089359de\" (UID: \"f054465e-403b-4d08-aa4d-b148089359de\") "
Oct 30 20:45:21.502069 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502038    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7\" (UniqueName: \"kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7\") pod \"7520efd8-ede8-4603-9b17-1f10fbd03119\" (UID: \"7520efd8-ede8-4603-9b17-1f10fbd03119\") "
Oct 30 20:45:21.502209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502031    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca" pod "d165444a-a39a-472d-9c93-520de9bda277" (UID: "d165444a-a39a-472d-9c93-520de9bda277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:21.502353 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502329    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca" pod "7520efd8-ede8-4603-9b17-1f10fbd03119" (UID: "7520efd8-ede8-4603-9b17-1f10fbd03119"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:21.502425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502381    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca" pod "f054465e-403b-4d08-aa4d-b148089359de" (UID: "f054465e-403b-4d08-aa4d-b148089359de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:21.502946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502922    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d165444a-a39a-472d-9c93-520de9bda277-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.503057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502951    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520efd8-ede8-4603-9b17-1f10fbd03119-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.503057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.502965    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f054465e-403b-4d08-aa4d-b148089359de-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.504147 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.504126    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h" pod "f054465e-403b-4d08-aa4d-b148089359de" (UID: "f054465e-403b-4d08-aa4d-b148089359de"). InnerVolumeSpecName "kube-api-access-mdr2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:21.504307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.504290    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7" pod "7520efd8-ede8-4603-9b17-1f10fbd03119" (UID: "7520efd8-ede8-4603-9b17-1f10fbd03119"). InnerVolumeSpecName "kube-api-access-dt9p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:21.504555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.504542    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx" pod "d165444a-a39a-472d-9c93-520de9bda277" (UID: "d165444a-a39a-472d-9c93-520de9bda277"). InnerVolumeSpecName "kube-api-access-w7wdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:21.543750 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.543723    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" path="/var/lib/kubelet/pods/07558179-69bc-45c3-b4cb-8191d8174c94/volumes"
Oct 30 20:45:21.569775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.569752    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"]
Oct 30 20:45:21.574302 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.574272    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-8n9t7"]
Oct 30 20:45:21.603912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.603881    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdr2h\" (UniqueName: \"kubernetes.io/projected/f054465e-403b-4d08-aa4d-b148089359de-kube-api-access-mdr2h\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.603912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.603903    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wdx\" (UniqueName: \"kubernetes.io/projected/d165444a-a39a-472d-9c93-520de9bda277-kube-api-access-w7wdx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.603912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.603913    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt9p7\" (UniqueName: \"kubernetes.io/projected/7520efd8-ede8-4603-9b17-1f10fbd03119-kube-api-access-dt9p7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.794244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.794158    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"]
Oct 30 20:45:21.798620 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:21.798597    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889885-5fssc"]
Oct 30 20:45:22.245134 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.245109    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"
Oct 30 20:45:22.245676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.245112    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm" event={"ID":"f054465e-403b-4d08-aa4d-b148089359de","Type":"ContainerDied","Data":"58260eb77f230fdfc4caf9fdeef56b021284dfedb4b7a71c7c298d1b96eccb35"}
Oct 30 20:45:22.245676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.245246    5967 scope.go:122] "RemoveContainer" containerID="d7b488cff5375710d4574b7af7bfc4acdbf956ddd28048c0e33c9cd91995bbf8"
Oct 30 20:45:22.246965 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.246914    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"
Oct 30 20:45:22.246965 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.247003    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp" event={"ID":"7520efd8-ede8-4603-9b17-1f10fbd03119","Type":"ContainerDied","Data":"1503b796a38d231b26f721a0e8249aed4ba3e33212c755e231dc9072dd75bc7c"}
Oct 30 20:45:22.250702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.250683    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"
Oct 30 20:45:22.250849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.250781    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l" event={"ID":"d165444a-a39a-472d-9c93-520de9bda277","Type":"ContainerDied","Data":"f0141b9c4fe83250b9c2f7dd3a76aaefb6d6ed7dc95e94fc1660d00ef97f7a9b"}
Oct 30 20:45:22.258055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.258035    5967 scope.go:122] "RemoveContainer" containerID="02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4"
Oct 30 20:45:22.267858 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.267835    5967 scope.go:122] "RemoveContainer" containerID="0cf22b3b9c7abce969253d31c10d0df7e998ddf70b7add283e35a8adb20b7bbf"
Oct 30 20:45:22.412598 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.412570    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"]
Oct 30 20:45:22.417097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.417075    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a78bf4-29889885-pszvp"]
Oct 30 20:45:22.525659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.525575    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"]
Oct 30 20:45:22.530093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.530073    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889885-pvjhm"]
Oct 30 20:45:22.646757 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.646709    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"]
Oct 30 20:45:22.648308 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:22.648276    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889885-rwp7l"]
Oct 30 20:45:23.544276 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:23.544244    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" path="/var/lib/kubelet/pods/7520efd8-ede8-4603-9b17-1f10fbd03119/volumes"
Oct 30 20:45:23.545048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:23.544654    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" path="/var/lib/kubelet/pods/7c734c66-ccde-4387-bb02-df9d805e817c/volumes"
Oct 30 20:45:23.545048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:23.545048    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" path="/var/lib/kubelet/pods/83b56850-03be-492d-a5fe-22dd65cdb8ee/volumes"
Oct 30 20:45:23.545762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:23.545721    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d165444a-a39a-472d-9c93-520de9bda277" path="/var/lib/kubelet/pods/d165444a-a39a-472d-9c93-520de9bda277/volumes"
Oct 30 20:45:23.546156 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:23.546138    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f054465e-403b-4d08-aa4d-b148089359de" path="/var/lib/kubelet/pods/f054465e-403b-4d08-aa4d-b148089359de/volumes"
Oct 30 20:45:25.047128 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:25.047097    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"]
Oct 30 20:45:25.051672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:25.051648    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca112033-rnbgx"]
Oct 30 20:45:25.544102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:25.544074    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb8a9ee-df3b-493f-a32f-c8acdaaa3128" path="/var/lib/kubelet/pods/6cb8a9ee-df3b-493f-a32f-c8acdaaa3128/volumes"
Oct 30 20:45:26.462526 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:26.462494    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:29.052778 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:29.052716    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"]
Oct 30 20:45:29.101487 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:29.101453    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2d71e2-7xfpk"]
Oct 30 20:45:29.544705 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:29.544671    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3be2dbe-c560-41c0-9f36-516ff41b4866" path="/var/lib/kubelet/pods/e3be2dbe-c560-41c0-9f36-516ff41b4866/volumes"
Oct 30 20:45:30.764828 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:30.764792    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:31.980891 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:31.980857    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:45:33.455489 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.455454    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"]
Oct 30 20:45:33.456112 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456098    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:33.456157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456113    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:33.456157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456127    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:33.456157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456132    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:33.456157 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456143    5967 cpu_manager.go:414] "containerMap: removing container" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456149    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456173    5967 cpu_manager.go:414] "containerMap: removing container" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456178    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456189    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456193    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456205    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456210    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456224    5967 cpu_manager.go:414] "containerMap: removing container" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456228    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456236    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456241    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456248    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456252    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456260    5967 cpu_manager.go:414] "containerMap: removing container" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456264    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456277    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456282    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456288    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456292    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456299    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456304    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456315    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456319    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456324    5967 cpu_manager.go:414] "containerMap: removing container" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456328    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456415    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456424    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456429    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456434    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456439    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456445    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456450    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456456    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456462    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456466    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456472    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456479    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7520efd8-ede8-4603-9b17-1f10fbd03119" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456483    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.456790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.456488    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c734c66-ccde-4387-bb02-df9d805e817c" containerName="cleanup"
Oct 30 20:45:33.462283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.462264    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.530765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.522104    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c650f2b7-openshift-default-dockercfg-whq9x\""
Oct 30 20:45:33.568771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.560259    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.569107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.569078    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7t6\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.569340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.569323    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.569475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.569459    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.570340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.569763    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.570340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.569898    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.570340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.569985    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.570340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.570095    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.570340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.570146    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.612902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.599093    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"]
Oct 30 20:45:33.675881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.675640    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.675881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.675828    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.675881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.675885    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.675927    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676039    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7t6\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676178    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676214    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676284    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676387    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676441    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676668    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.676845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.676718    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.677022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.677003    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.677094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.677067    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.679964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.679935    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.682402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.682378    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.734950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.734888    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.739154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.739116    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7t6\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6\") pod \"e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") " pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:33.776827 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:33.776785    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:34.227026 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:34.227002    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4947cbd_7f16_417b_9730_8b226f74a4f7.slice/crio-67f318abdfed86a96457a986eca7f7c0b79fc2cc04a139b3c2c071deff8aa934 WatchSource:0}: Error finding container 67f318abdfed86a96457a986eca7f7c0b79fc2cc04a139b3c2c071deff8aa934: Status 404 returned error can't find the container with id 67f318abdfed86a96457a986eca7f7c0b79fc2cc04a139b3c2c071deff8aa934
Oct 30 20:45:34.227295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.227274    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"]
Oct 30 20:45:34.229225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.229195    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:34.229298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.229259    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:34.229298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.229285    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:34.360087 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.360046    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" event={"ID":"a4947cbd-7f16-417b-9730-8b226f74a4f7","Type":"ContainerStarted","Data":"b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"}
Oct 30 20:45:34.360266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.360122    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" event={"ID":"a4947cbd-7f16-417b-9730-8b226f74a4f7","Type":"ContainerStarted","Data":"67f318abdfed86a96457a986eca7f7c0b79fc2cc04a139b3c2c071deff8aa934"}
Oct 30 20:45:34.482155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.482055    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podStartSLOduration=1.48204138 podStartE2EDuration="1.48204138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:34.477094265 +0000 UTC m=+1515.077896226" watchObservedRunningTime="2026-10-30 20:45:34.48204138 +0000 UTC m=+1515.082843340"
Oct 30 20:45:34.777875 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:34.777808    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:35.049362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.049282    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"]
Oct 30 20:45:35.053493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.053467    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9062b2-dmxtb"]
Oct 30 20:45:35.543436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.543407    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0d3205-3118-4584-91e4-aabc62e29b12" path="/var/lib/kubelet/pods/0c0d3205-3118-4584-91e4-aabc62e29b12/volumes"
Oct 30 20:45:35.783795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.783727    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:35.889819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.889682    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"]
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.890711    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.890731    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.890776    5967 cpu_manager.go:414] "containerMap: removing container" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.890784    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.890803    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.890811    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.891107    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="07558179-69bc-45c3-b4cb-8191d8174c94" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.891129    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d165444a-a39a-472d-9c93-520de9bda277" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.891149    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b56850-03be-492d-a5fe-22dd65cdb8ee" containerName="cleanup"
Oct 30 20:45:35.904525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.891164    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f054465e-403b-4d08-aa4d-b148089359de" containerName="cleanup"
Oct 30 20:45:35.928549 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.928516    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:35.977886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:35.977789    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"]
Oct 30 20:45:36.033421 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.033354    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.033829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.033786    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkf5d\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.034075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.034059    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.034233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.034216    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.034501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.034486    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.035475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.035431    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.035718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.035703    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.035956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.035940    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.036114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.036101    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.048101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.040986    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d71c23-openshift-default-dockercfg-tcbdc\""
Oct 30 20:45:36.138003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.137963    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138020    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138049    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138093    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138130    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138319    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138345    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkf5d\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138413    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138446    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138504    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138720 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138700    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138866 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138848    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.138901 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.138885    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.139030 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.139009    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.141143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.141074    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.141372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.141273    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.196540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.196513    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkf5d\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.196696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.196602    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token\") pod \"e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") " pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.355546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.355516    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:36.382682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.382657    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:36.383889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.383868    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:45:36.562597 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:36.562564    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:36.805820 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:45:36.805794    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994aa1e0_293c_4c27_9d9f_b402607af934.slice/crio-ad258ba51d65cb121754b59097b057775ab77d0272e40df64c3deca1cb961ed9 WatchSource:0}: Error finding container ad258ba51d65cb121754b59097b057775ab77d0272e40df64c3deca1cb961ed9: Status 404 returned error can't find the container with id ad258ba51d65cb121754b59097b057775ab77d0272e40df64c3deca1cb961ed9
Oct 30 20:45:36.806728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.806704    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"]
Oct 30 20:45:36.808814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.808783    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:36.808922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.808853    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:36.808922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:36.808899    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:37.051607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:37.051569    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"]
Oct 30 20:45:37.055723 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:37.055665    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d6b810-mcll2"]
Oct 30 20:45:37.392808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:37.392688    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" event={"ID":"994aa1e0-293c-4c27-9d9f-b402607af934","Type":"ContainerStarted","Data":"7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"}
Oct 30 20:45:37.392808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:37.392760    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" event={"ID":"994aa1e0-293c-4c27-9d9f-b402607af934","Type":"ContainerStarted","Data":"ad258ba51d65cb121754b59097b057775ab77d0272e40df64c3deca1cb961ed9"}
Oct 30 20:45:37.512480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:37.512428    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podStartSLOduration=2.512413983 podStartE2EDuration="2.512413983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:37.510660426 +0000 UTC m=+1518.111462386" watchObservedRunningTime="2026-10-30 20:45:37.512413983 +0000 UTC m=+1518.113215943"
Oct 30 20:45:37.544957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:37.544928    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de132c84-f27e-432c-b3ff-bbfbc8877537" path="/var/lib/kubelet/pods/de132c84-f27e-432c-b3ff-bbfbc8877537/volumes"
Oct 30 20:45:38.356493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:38.356433    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:38.363145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:38.363117    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:38.404769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:38.402568    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:38.404769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:38.403364    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:45:45.840141 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:45.840112    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:46.578843 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:46.578801    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:50.053061 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:50.053030    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"]
Oct 30 20:45:50.101568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:50.101537    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44ba5ff-7s9r2"]
Oct 30 20:45:51.543279 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:51.543247    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b746798-4a6f-473d-b0e5-3fc86018b23e" path="/var/lib/kubelet/pods/1b746798-4a6f-473d-b0e5-3fc86018b23e/volumes"
Oct 30 20:45:54.053256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:54.053221    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"]
Oct 30 20:45:54.102521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:54.102489    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c80311-66g7n"]
Oct 30 20:45:55.049143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:55.049113    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"]
Oct 30 20:45:55.099393 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:55.099361    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3257db-zplmq"]
Oct 30 20:45:55.544099 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:55.544071    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d183b1af-5f73-45ad-907f-1e3fd2f60ee9" path="/var/lib/kubelet/pods/d183b1af-5f73-45ad-907f-1e3fd2f60ee9/volumes"
Oct 30 20:45:55.544551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:55.544533    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4" path="/var/lib/kubelet/pods/f49a6f89-4b5f-4fb1-bbcb-b40f448a27e4/volumes"
Oct 30 20:45:56.657860 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:45:56.657831    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:58.056781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:58.056736    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"]
Oct 30 20:45:58.106196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:58.106169    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db6080c-tvj5p"]
Oct 30 20:45:59.051301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.051232    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"]
Oct 30 20:45:59.100306 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.100277    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"]
Oct 30 20:45:59.104095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.104082    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000a85ca-r9tdc"]
Oct 30 20:45:59.107307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.107285    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c779eb29-ptzk4"]
Oct 30 20:45:59.544391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.544360    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589cb23b-c231-4ee3-a715-80f021c382c2" path="/var/lib/kubelet/pods/589cb23b-c231-4ee3-a715-80f021c382c2/volumes"
Oct 30 20:45:59.544860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.544845    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3414b8-48e9-4417-865e-66209cb49a7b" path="/var/lib/kubelet/pods/dc3414b8-48e9-4417-865e-66209cb49a7b/volumes"
Oct 30 20:45:59.545252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:45:59.545240    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8cfc00f-a341-4d43-84eb-fff343200ef9" path="/var/lib/kubelet/pods/e8cfc00f-a341-4d43-84eb-fff343200ef9/volumes"
Oct 30 20:46:00.764788 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:00.763605    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:46:01.980412 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:01.980371    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:46:02.047589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:02.047562    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"]
Oct 30 20:46:02.096967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:02.096943    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c41545d2-5tfl5"]
Oct 30 20:46:03.046946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:03.046912    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"]
Oct 30 20:46:03.096107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:03.096083    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cffb967e-ns9rq"]
Oct 30 20:46:03.544596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:03.544564    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765516f0-185e-4e1a-a7df-5b374dccc564" path="/var/lib/kubelet/pods/765516f0-185e-4e1a-a7df-5b374dccc564/volumes"
Oct 30 20:46:03.545028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:03.545014    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e13fb3-a54b-43fd-8474-c570a30c9e02" path="/var/lib/kubelet/pods/e0e13fb3-a54b-43fd-8474-c570a30c9e02/volumes"
Oct 30 20:46:04.047663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:04.047626    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"]
Oct 30 20:46:04.096541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:04.096509    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156ebb11-xjpns"]
Oct 30 20:46:05.177616 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.177575    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"]
Oct 30 20:46:05.183949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.183924    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.239360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.239336    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"]
Oct 30 20:46:05.269485 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.269449    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.269659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.269494    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56z8s\" (UniqueName: \"kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.269659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.269628    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.282863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.282836    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:05.282863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.282836    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:46:05.283072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.282928    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:46:05.284752 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.284708    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:46:05.371056 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.371029    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.371236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.371222    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.371280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.371246    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56z8s\" (UniqueName: \"kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.371835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.371818    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.373505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.373488    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.427179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.427158    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56z8s\" (UniqueName: \"kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s\") pod \"maas-api-revoke-keys-e2e-ait-0d757729-qt88p\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.500262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.500241    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:05.546788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.546719    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb3cb89-f75c-48aa-b308-bdf782704ae2" path="/var/lib/kubelet/pods/bfb3cb89-f75c-48aa-b308-bdf782704ae2/volumes"
Oct 30 20:46:05.884693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:05.884662    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"]
Oct 30 20:46:06.676118 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:06.676076    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p" event={"ID":"0a28aec6-96d9-4637-a8b4-2cd549d4485e","Type":"ContainerStarted","Data":"0113fb3a13bced40ec2de394a87ff6599a6c9b3b7272dd6c6843abaf5be8d16a"}
Oct 30 20:46:06.676785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:06.676758    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p" event={"ID":"0a28aec6-96d9-4637-a8b4-2cd549d4485e","Type":"ContainerStarted","Data":"7c98bbdc3512451663a8d50a12767c55e3eacf40178c4f4bd4fd7e8dba65a0f6"}
Oct 30 20:46:06.861312 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:06.861273    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:46:07.440646 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:07.440598    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:46:07.441127 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:07.441094    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:46:07.686889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:07.686853    5967 generic.go:356] "Generic (PLEG): container finished" podID="0a28aec6-96d9-4637-a8b4-2cd549d4485e" containerID="0113fb3a13bced40ec2de394a87ff6599a6c9b3b7272dd6c6843abaf5be8d16a" exitCode=0
Oct 30 20:46:07.687328 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:07.686937    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p" event={"ID":"0a28aec6-96d9-4637-a8b4-2cd549d4485e","Type":"ContainerDied","Data":"0113fb3a13bced40ec2de394a87ff6599a6c9b3b7272dd6c6843abaf5be8d16a"}
Oct 30 20:46:08.862187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.862164    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:08.931681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.931649    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca\") pod \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") "
Oct 30 20:46:08.931836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.931702    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s\" (UniqueName: \"kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s\") pod \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") "
Oct 30 20:46:08.932057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.932041    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca\") pod \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\" (UID: \"0a28aec6-96d9-4637-a8b4-2cd549d4485e\") "
Oct 30 20:46:08.932590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.932563    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca" pod "0a28aec6-96d9-4637-a8b4-2cd549d4485e" (UID: "0a28aec6-96d9-4637-a8b4-2cd549d4485e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:08.933968 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.933946    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca" pod "0a28aec6-96d9-4637-a8b4-2cd549d4485e" (UID: "0a28aec6-96d9-4637-a8b4-2cd549d4485e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:08.934074 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:08.933978    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s" pod "0a28aec6-96d9-4637-a8b4-2cd549d4485e" (UID: "0a28aec6-96d9-4637-a8b4-2cd549d4485e"). InnerVolumeSpecName "kube-api-access-56z8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:09.034174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:09.034144    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:09.034379 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:09.034353    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28aec6-96d9-4637-a8b4-2cd549d4485e-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:09.034487 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:09.034381    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56z8s\" (UniqueName: \"kubernetes.io/projected/0a28aec6-96d9-4637-a8b4-2cd549d4485e-kube-api-access-56z8s\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:09.707281 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:09.707257    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"
Oct 30 20:46:09.707455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:09.707296    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p" event={"ID":"0a28aec6-96d9-4637-a8b4-2cd549d4485e","Type":"ContainerDied","Data":"7c98bbdc3512451663a8d50a12767c55e3eacf40178c4f4bd4fd7e8dba65a0f6"}
Oct 30 20:46:09.707455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:09.707345    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c98bbdc3512451663a8d50a12767c55e3eacf40178c4f4bd4fd7e8dba65a0f6"
Oct 30 20:46:10.056960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:10.056894    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"]
Oct 30 20:46:10.061022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:10.060993    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e190cd35-mvpsw"]
Oct 30 20:46:11.059463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:11.059429    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"]
Oct 30 20:46:11.108270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:11.108242    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfe83f4-2z4lh"]
Oct 30 20:46:11.544052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:11.544023    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cd695e-2b63-45e2-bd97-ec9c487f6abd" path="/var/lib/kubelet/pods/82cd695e-2b63-45e2-bd97-ec9c487f6abd/volumes"
Oct 30 20:46:11.544461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:11.544447    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47b7017-aa1e-4fae-a439-5b0f5d43d664" path="/var/lib/kubelet/pods/e47b7017-aa1e-4fae-a439-5b0f5d43d664/volumes"
Oct 30 20:46:12.125616 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.125588    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"]
Oct 30 20:46:12.126272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.126258    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0a28aec6-96d9-4637-a8b4-2cd549d4485e" containerName="revoke-keys"
Oct 30 20:46:12.126323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.126273    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a28aec6-96d9-4637-a8b4-2cd549d4485e" containerName="revoke-keys"
Oct 30 20:46:12.126386 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.126377    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a28aec6-96d9-4637-a8b4-2cd549d4485e" containerName="revoke-keys"
Oct 30 20:46:12.131153 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.131130    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.183164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.183140    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:46:12.232650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.232618    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:12.232650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.232633    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:46:12.232895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.232670    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:46:12.237345 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.237312    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"]
Oct 30 20:46:12.276655 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.276617    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.276882 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.276832    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5b7\" (UniqueName: \"kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.277054 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.277028    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.378120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.378049    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5b7\" (UniqueName: \"kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.378238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.378159    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.378238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.378215    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.378902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.378883    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.380443 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.380422    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.435490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.435456    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5b7\" (UniqueName: \"kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7\") pod \"maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.445580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.445552    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:12.828999 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:12.828959    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"]
Oct 30 20:46:12.832735 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:46:12.832335    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc425ce_c28a_43e1_93a5_e4263f17f903.slice/crio-b523d18f1cd88a55a4b84ef5739b903f0abde22f085fa21b140ace11f7fb42d6 WatchSource:0}: Error finding container b523d18f1cd88a55a4b84ef5739b903f0abde22f085fa21b140ace11f7fb42d6: Status 404 returned error can't find the container with id b523d18f1cd88a55a4b84ef5739b903f0abde22f085fa21b140ace11f7fb42d6
Oct 30 20:46:13.742726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:13.742689    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb" event={"ID":"0bc425ce-c28a-43e1-93a5-e4263f17f903","Type":"ContainerStarted","Data":"fa0761b6b56d2fe5af636e83fa239cb817f109ea433f6e6e4e4e36539d0c3959"}
Oct 30 20:46:13.743114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:13.742791    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb" event={"ID":"0bc425ce-c28a-43e1-93a5-e4263f17f903","Type":"ContainerStarted","Data":"b523d18f1cd88a55a4b84ef5739b903f0abde22f085fa21b140ace11f7fb42d6"}
Oct 30 20:46:13.858478 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:13.858435    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb" podStartSLOduration=1.858421866 podStartE2EDuration="1.858421866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:13.858025964 +0000 UTC m=+1554.458827920" watchObservedRunningTime="2026-10-30 20:46:13.858421866 +0000 UTC m=+1554.459223824"
Oct 30 20:46:14.753240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:14.753202    5967 generic.go:356] "Generic (PLEG): container finished" podID="0bc425ce-c28a-43e1-93a5-e4263f17f903" containerID="fa0761b6b56d2fe5af636e83fa239cb817f109ea433f6e6e4e4e36539d0c3959" exitCode=0
Oct 30 20:46:14.753616 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:14.753283    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb" event={"ID":"0bc425ce-c28a-43e1-93a5-e4263f17f903","Type":"ContainerDied","Data":"fa0761b6b56d2fe5af636e83fa239cb817f109ea433f6e6e4e4e36539d0c3959"}
Oct 30 20:46:15.162175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.162088    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"]
Oct 30 20:46:15.166625 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.166602    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.219789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.219720    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27219c73-openshift-default-dockercfg-8npnb\""
Oct 30 20:46:15.232512 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.232483    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"]
Oct 30 20:46:15.318955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.318918    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz79g\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.318996    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319081    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319158 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319119    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319327 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319253    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319456    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319543 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319517    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319542    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.319603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.319583    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421418    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz79g\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421494    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421545    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421585    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421685    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421943    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.421996    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.422021    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.422062    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.422608    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.423815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.423191    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.441765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.441161    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.441954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.441814    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.446807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.442465    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.473782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.471060    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.500798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.489297    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.500798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.494219    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.500798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.494383    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz79g\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g\") pod \"e2e-mt-27219c73-openshift-default-88d599849-rnbxz\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") " pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.781287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.781255    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:15.845606 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:15.845090    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:46:15.958033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:15.958009    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:16.030419 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.030338    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca\") pod \"0bc425ce-c28a-43e1-93a5-e4263f17f903\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") "
Oct 30 20:46:16.030653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.030635    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7\" (UniqueName: \"kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7\") pod \"0bc425ce-c28a-43e1-93a5-e4263f17f903\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") "
Oct 30 20:46:16.030788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.030675    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca\") pod \"0bc425ce-c28a-43e1-93a5-e4263f17f903\" (UID: \"0bc425ce-c28a-43e1-93a5-e4263f17f903\") "
Oct 30 20:46:16.030788 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.030690    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca" pod "0bc425ce-c28a-43e1-93a5-e4263f17f903" (UID: "0bc425ce-c28a-43e1-93a5-e4263f17f903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:16.031566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.031545    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:16.032869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.032840    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca" pod "0bc425ce-c28a-43e1-93a5-e4263f17f903" (UID: "0bc425ce-c28a-43e1-93a5-e4263f17f903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:16.032963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.032916    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7" pod "0bc425ce-c28a-43e1-93a5-e4263f17f903" (UID: "0bc425ce-c28a-43e1-93a5-e4263f17f903"). InnerVolumeSpecName "kube-api-access-bk5b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:16.132539 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.132504    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5b7\" (UniqueName: \"kubernetes.io/projected/0bc425ce-c28a-43e1-93a5-e4263f17f903-kube-api-access-bk5b7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:16.132539 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.132533    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc425ce-c28a-43e1-93a5-e4263f17f903-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:16.241731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.241707    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"]
Oct 30 20:46:16.242313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:46:16.242275    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae17a91b_252c_4bce_a0ad_a0e5d2ee9f4d.slice/crio-f0ca1176e6a1c9a7d3a35b832d7253fbbbe861b9f0fe56d3e8ad6b988be7db44 WatchSource:0}: Error finding container f0ca1176e6a1c9a7d3a35b832d7253fbbbe861b9f0fe56d3e8ad6b988be7db44: Status 404 returned error can't find the container with id f0ca1176e6a1c9a7d3a35b832d7253fbbbe861b9f0fe56d3e8ad6b988be7db44
Oct 30 20:46:16.244939 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.244899    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:16.245048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.244992    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:16.245048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.245035    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:16.782015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.781967    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb" event={"ID":"0bc425ce-c28a-43e1-93a5-e4263f17f903","Type":"ContainerDied","Data":"b523d18f1cd88a55a4b84ef5739b903f0abde22f085fa21b140ace11f7fb42d6"}
Oct 30 20:46:16.782642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.782072    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b523d18f1cd88a55a4b84ef5739b903f0abde22f085fa21b140ace11f7fb42d6"
Oct 30 20:46:16.782642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.782191    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"
Oct 30 20:46:16.793447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.793405    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" event={"ID":"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d","Type":"ContainerStarted","Data":"14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"}
Oct 30 20:46:16.793671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.793496    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" event={"ID":"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d","Type":"ContainerStarted","Data":"f0ca1176e6a1c9a7d3a35b832d7253fbbbe861b9f0fe56d3e8ad6b988be7db44"}
Oct 30 20:46:16.878609 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:16.878583    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520efd8_ede8_4603_9b17_1f10fbd03119.slice/crio-conmon-02729aa5858316296628f70701cc68be1ac41a7c11cc217f9fb2e43e415c32d4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:46:16.914200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:16.914154    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podStartSLOduration=1.914140609 podStartE2EDuration="1.914140609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:16.912661066 +0000 UTC m=+1557.513463036" watchObservedRunningTime="2026-10-30 20:46:16.914140609 +0000 UTC m=+1557.514942569"
Oct 30 20:46:17.782020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:17.781989    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:17.789395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:17.789365    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:17.801502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:17.801475    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:17.802277 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:17.802257    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:46:18.049897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:18.049789    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"]
Oct 30 20:46:18.054208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:18.054181    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68525d2e-4fzfn"]
Oct 30 20:46:19.050643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:19.050614    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"]
Oct 30 20:46:19.054276 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:19.054255    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5f3a1e-g4gjd"]
Oct 30 20:46:19.544421 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:19.544391    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0553767a-9895-4946-b66c-516203b38291" path="/var/lib/kubelet/pods/0553767a-9895-4946-b66c-516203b38291/volumes"
Oct 30 20:46:19.544842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:19.544829    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db77f87-7883-4939-980c-6d38a4fa91cd" path="/var/lib/kubelet/pods/1db77f87-7883-4939-980c-6d38a4fa91cd/volumes"
Oct 30 20:46:20.061121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:20.061085    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"]
Oct 30 20:46:20.110109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:20.110077    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5804185-8kvgq"]
Oct 30 20:46:20.308384 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:20.308349    5967 scope.go:122] "RemoveContainer" containerID="15e2faff4291e9973cf9ef9035ff0a1322fcee123f12db1e3e9242d19c42adc7"
Oct 30 20:46:20.319966 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:20.319947    5967 scope.go:122] "RemoveContainer" containerID="9f066adc0a68c072f51bb87b4fcdb7e5672b853ff6396ffc42f96b2491e1cfef"
Oct 30 20:46:20.330285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:20.330264    5967 scope.go:122] "RemoveContainer" containerID="15a92de5edfd6d5d0972b77e022fa385befd8732865db223a102214d4b887645"
Oct 30 20:46:20.340883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:20.340859    5967 scope.go:122] "RemoveContainer" containerID="fc80cde67dfcb63634d77330ed8f9d683ea32510e0f5d213c1c5269833d62185"
Oct 30 20:46:21.544209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:21.544177    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1937f6d0-aa8f-4061-8e22-9291b1746c50" path="/var/lib/kubelet/pods/1937f6d0-aa8f-4061-8e22-9291b1746c50/volumes"
Oct 30 20:46:22.050676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:22.050645    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"]
Oct 30 20:46:22.099762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:22.099701    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5f06e1-2rl5r"]
Oct 30 20:46:23.545583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:23.545541    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010a8f71-8fcf-4c29-88a7-5887f1f08f92" path="/var/lib/kubelet/pods/010a8f71-8fcf-4c29-88a7-5887f1f08f92/volumes"
Oct 30 20:46:24.054558 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:24.054526    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"]
Oct 30 20:46:24.103065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:24.103030    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dccfc9e-mwg2r"]
Oct 30 20:46:25.545009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:25.544982    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4579070e-3803-49b0-b2dc-4cbd9f726f7e" path="/var/lib/kubelet/pods/4579070e-3803-49b0-b2dc-4cbd9f726f7e/volumes"
Oct 30 20:46:29.049065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:29.049006    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"]
Oct 30 20:46:29.098321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:29.098292    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e99ced4-blwcm"]
Oct 30 20:46:29.553445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:29.553405    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd62776-1b2f-4592-87ee-c8846599692f" path="/var/lib/kubelet/pods/fdd62776-1b2f-4592-87ee-c8846599692f/volumes"
Oct 30 20:46:30.049493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:30.049461    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"]
Oct 30 20:46:30.097262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:30.097240    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1af342-9kvsm"]
Oct 30 20:46:31.047034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:31.047003    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"]
Oct 30 20:46:31.095674 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:31.095642    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d113c27-k4j5s"]
Oct 30 20:46:31.554792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:31.554733    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf07786d-5756-4ace-ab26-9081a682d12e" path="/var/lib/kubelet/pods/bf07786d-5756-4ace-ab26-9081a682d12e/volumes"
Oct 30 20:46:31.555493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:31.555451    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0348ff2-00b0-4c8d-ae61-0540ac075b2a" path="/var/lib/kubelet/pods/e0348ff2-00b0-4c8d-ae61-0540ac075b2a/volumes"
Oct 30 20:46:31.980624 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:46:31.980594    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:46:34.048246 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:34.048213    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"]
Oct 30 20:46:34.096787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:34.096723    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71882aa2-x74bt"]
Oct 30 20:46:35.544705 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:35.544677    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe044cf0-5710-4332-b482-b11b4e03be39" path="/var/lib/kubelet/pods/fe044cf0-5710-4332-b482-b11b4e03be39/volumes"
Oct 30 20:46:40.048014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:40.047983    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"]
Oct 30 20:46:40.097657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:40.097627    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6080963-vwlmn"]
Oct 30 20:46:41.544416 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:41.544386    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d634662-f0c0-47d4-b9d1-c001173afcc0" path="/var/lib/kubelet/pods/2d634662-f0c0-47d4-b9d1-c001173afcc0/volumes"
Oct 30 20:46:51.047406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:51.047375    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"]
Oct 30 20:46:51.096540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:51.096514    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c55f38-6bvp5"]
Oct 30 20:46:51.544332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:51.544300    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3384efb9-738f-4d80-950b-dbdf4a15010d" path="/var/lib/kubelet/pods/3384efb9-738f-4d80-950b-dbdf4a15010d/volumes"
Oct 30 20:46:59.304256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:59.304221    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"]
Oct 30 20:46:59.352861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:59.352830    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d757729-qt88p"]
Oct 30 20:46:59.544401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:46:59.544367    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a28aec6-96d9-4637-a8b4-2cd549d4485e" path="/var/lib/kubelet/pods/0a28aec6-96d9-4637-a8b4-2cd549d4485e/volumes"
Oct 30 20:47:00.413306 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:00.413277    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"]
Oct 30 20:47:00.462361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:00.462330    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bdb58a-6jdcb"]
Oct 30 20:47:01.544470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:01.544428    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc425ce-c28a-43e1-93a5-e4263f17f903" path="/var/lib/kubelet/pods/0bc425ce-c28a-43e1-93a5-e4263f17f903/volumes"
Oct 30 20:47:02.130584 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:47:02.130524    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:47:15.048127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:15.048091    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"]
Oct 30 20:47:15.097147 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:15.097113    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0d4bcf-tzqd9"]
Oct 30 20:47:15.544883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:15.544851    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147623a5-a309-4a72-a70e-88fe59456ef5" path="/var/lib/kubelet/pods/147623a5-a309-4a72-a70e-88fe59456ef5/volumes"
Oct 30 20:47:16.569286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.569253    5967 generic.go:356] "Generic (PLEG): container finished" podID="994aa1e0-293c-4c27-9d9f-b402607af934" containerID="7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf" exitCode=0
Oct 30 20:47:16.569724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.569329    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" event={"ID":"994aa1e0-293c-4c27-9d9f-b402607af934","Type":"ContainerDied","Data":"7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"}
Oct 30 20:47:16.569844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.569775    5967 scope.go:122] "RemoveContainer" containerID="7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"
Oct 30 20:47:16.571207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.571186    5967 generic.go:356] "Generic (PLEG): container finished" podID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerID="b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4" exitCode=0
Oct 30 20:47:16.571310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.571250    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" event={"ID":"a4947cbd-7f16-417b-9730-8b226f74a4f7","Type":"ContainerDied","Data":"b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"}
Oct 30 20:47:16.571356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.571312    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:16.571391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.571374    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:16.571423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.571415    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:16.571545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.571533    5967 scope.go:122] "RemoveContainer" containerID="b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"
Oct 30 20:47:16.572876 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.572850    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:16.572985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.572908    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:16.572985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:16.572938    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:17.633821 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:17.633789    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" event={"ID":"994aa1e0-293c-4c27-9d9f-b402607af934","Type":"ContainerStarted","Data":"1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6"}
Oct 30 20:47:17.636429 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:17.636403    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" event={"ID":"a4947cbd-7f16-417b-9730-8b226f74a4f7","Type":"ContainerStarted","Data":"88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff"}
Oct 30 20:47:17.804766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:17.798588    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:47:17.865075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:17.864278    5967 patch_prober.go:29] interesting pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:47:17.865075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:17.864334    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 20:47:18.364767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:18.364436    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:47:18.393765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:18.373669    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:47:18.393765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:18.373725    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:47:18.810767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:18.797217    5967 patch_prober.go:29] interesting pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:47:18.810767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:18.797277    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 20:47:19.356070 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:19.356031    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:47:19.356252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:19.356085    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:47:19.792765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:19.782056    5967 patch_prober.go:29] interesting pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:47:19.792765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:19.782111    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 20:47:20.356597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.356569    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:47:20.357113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.357082    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:47:20.429650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.429618    5967 scope.go:122] "RemoveContainer" containerID="b8545228cc14ab34a44d8794ed0cc69af615d606c121a22b5f0b0e22c5493c15"
Oct 30 20:47:20.442279 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.442260    5967 scope.go:122] "RemoveContainer" containerID="bae2f9738e8e0bbb79bdaabd9f63b958e8e6870ba57e8a36a74ef7af1e19c5be"
Oct 30 20:47:20.452780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.452715    5967 scope.go:122] "RemoveContainer" containerID="77d9c9c71936cec57dedacf6852a5492bb36914013cd28afbe478e2d94390e3e"
Oct 30 20:47:20.463139 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.463124    5967 scope.go:122] "RemoveContainer" containerID="10a942ee82bb97cfebe916bd18fa6f0d2d408be14b2535c85452e9f47b186f02"
Oct 30 20:47:20.497462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.497442    5967 scope.go:122] "RemoveContainer" containerID="63da20a0608ecffd7b023baa587c061d3e1781c6d0d079ef351e16f07f8b8e4c"
Oct 30 20:47:20.536805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.536593    5967 scope.go:122] "RemoveContainer" containerID="17e8dd575441237a255ac3205e5ca01778252a4f66c410e03e9ae617e2029001"
Oct 30 20:47:20.548217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.547622    5967 scope.go:122] "RemoveContainer" containerID="9f1ef5bef7f2f35fb9514392f6b79ed071e8da33e1fe2e9a3b5ebe81acb376d6"
Oct 30 20:47:20.557683 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.557667    5967 scope.go:122] "RemoveContainer" containerID="b44087b83456fce8ada2b4f93a0560708afc8d74c0423507ffe740a2ba91cc53"
Oct 30 20:47:20.567895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.567875    5967 scope.go:122] "RemoveContainer" containerID="b612cc705855e85a11a3030e640d76284064604c0d197d86f2b9118d9dc0507b"
Oct 30 20:47:20.629185 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.629161    5967 scope.go:122] "RemoveContainer" containerID="9097187f48ff5a84127db4086777f8e4cd1f44f815e1fa0e2100c5013121b658"
Oct 30 20:47:20.641812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.641792    5967 scope.go:122] "RemoveContainer" containerID="f1eb1c51381a231108c2f05c9a32ef54503754ecc16b90814d65cc22d1c2b387"
Oct 30 20:47:20.654880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.654857    5967 scope.go:122] "RemoveContainer" containerID="4bc7674a60d04917a565cac604b18d2b68cf46e6bfed8e59b5cdf736f107b83b"
Oct 30 20:47:20.665650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.665632    5967 scope.go:122] "RemoveContainer" containerID="b79cbe1db7b7fdf8791786e8e262eef1161015fd5cdcbff40eb2c516d6b34a13"
Oct 30 20:47:20.731784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.731697    5967 scope.go:122] "RemoveContainer" containerID="1a42d076d4339bcf6c478fa171200a54b540554fdd0a0b6437ddf5c6808fc08c"
Oct 30 20:47:20.743751 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.743720    5967 scope.go:122] "RemoveContainer" containerID="47fb32ca4257446a11df93db028d292b3e3a777d97f9fe72ffcc274972ef1599"
Oct 30 20:47:20.753491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.753475    5967 scope.go:122] "RemoveContainer" containerID="002775af1fde945f245ec8062df99df60639ece20ad4cf774a561c4646c529d4"
Oct 30 20:47:20.764028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.764007    5967 scope.go:122] "RemoveContainer" containerID="fbbe7aa7a4f78addcf8a9585bf0f20f95a3d053d0cd0ddb616ea1600488e2869"
Oct 30 20:47:20.784800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.780076    5967 patch_prober.go:29] interesting pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:47:20.784800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.780140    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 20:47:20.825773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.825719    5967 scope.go:122] "RemoveContainer" containerID="f30a92664b877d861d7ced349eceb4d5516302c0bbbfaf8c2c9aef88e9e6e918"
Oct 30 20:47:20.837119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.837101    5967 scope.go:122] "RemoveContainer" containerID="fabe892216ab397e17cb8d33fd675c3322bb28c69bd413278380457a3b6ba865"
Oct 30 20:47:20.846776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:20.846756    5967 scope.go:122] "RemoveContainer" containerID="4df9b65067dffedf2ca06ccde7caa09d24111af11d0df1c4601dfe94c82c5607"
Oct 30 20:47:21.356417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:21.356386    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:47:21.356597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:21.356429    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:47:21.393770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:21.385839    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:47:21.786768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:21.783502    5967 patch_prober.go:29] interesting pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:47:21.786768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:21.783561    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 20:47:22.356022 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:22.355992    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:47:22.356282 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:22.356036    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:47:22.777909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:22.777870    5967 patch_prober.go:29] interesting pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:47:22.778330 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:22.777943    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 20:47:23.360462 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:23.360435    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:47:23.360913 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:23.360893    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:47:23.361612 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:23.361593    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:47:23.783033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:23.783000    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:47:23.784024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:23.784003    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:47:24.737359 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.737326    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"]
Oct 30 20:47:24.737571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.737534    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" containerID="cri-o://f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156" gracePeriod=30
Oct 30 20:47:24.787883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.787861    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"]
Oct 30 20:47:24.788286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.788040    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy" containerID="cri-o://20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f" gracePeriod=30
Oct 30 20:47:24.792024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.791995    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"]
Oct 30 20:47:24.792242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.792217    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy" containerID="cri-o://21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7" gracePeriod=30
Oct 30 20:47:24.795175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.795150    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"]
Oct 30 20:47:24.795302 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:24.795285    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy" containerID="cri-o://02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a" gracePeriod=30
Oct 30 20:47:29.499377 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:29.499301    5967 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:47:29.499377 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:29.499366    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:47:30.130258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.130225    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:47:30.134399 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.134377    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:47:30.166910 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.166885    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:47:30.175951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.175932    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:47:30.256604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256578    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.256798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256668    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.256798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256760    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.256921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256818    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.256921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256875    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.256921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256918    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.257082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256926    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.257082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.256981    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.257082 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257061    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.257229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257099    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257113    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.257229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257150    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.257229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257153    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.257229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257188    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token\") pod \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\" (UID: \"4c6bd0c7-1900-429e-aa72-55bacb5f12b2\") "
Oct 30 20:47:30.257229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257209    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257235    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257306    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257334    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257338    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257394    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257428    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257536    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257597    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data\") pod \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\" (UID: \"185df700-2f7c-4ff9-af4e-66a44b5bca2b\") "
Oct 30 20:47:30.257976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257649    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:30.257976 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.257876    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258572    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258596    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258613    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258627    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258640    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258643    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258655    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258670    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.258944    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.259089 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.259025    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.259956 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.259927    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.260430 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.260406    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:30.260710 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.260686    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.260861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.260787    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "kube-api-access-v927v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.261010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.260992    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.261529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.261509    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo" pod "185df700-2f7c-4ff9-af4e-66a44b5bca2b" (UID: "185df700-2f7c-4ff9-af4e-66a44b5bca2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:30.261648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.261627    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "kube-api-access-6h6wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.261829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.261808    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token" pod "4c6bd0c7-1900-429e-aa72-55bacb5f12b2" (UID: "4c6bd0c7-1900-429e-aa72-55bacb5f12b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.360008 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.359911    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.360008 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.359957    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.360008 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.359993    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360093    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.360293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360131    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360164    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360185    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.360293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360202    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.360293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360244    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360276    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360343    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360368    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360402    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360438    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360482    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360523    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360528    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.360609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360606    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.361103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360672    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.361103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360704    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.361103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360758    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data\") pod \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\" (UID: \"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232\") "
Oct 30 20:47:30.361103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360800    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889\") pod \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\" (UID: \"a4d3ffe8-d8ed-4425-bf2d-824bb41af110\") "
Oct 30 20:47:30.361103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.360868    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:30.361345 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361321    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.361528 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361510    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.361824 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361799    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361827    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361843    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6wq\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-kube-api-access-6h6wq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361856    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361870    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361886    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361901    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.361917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361914    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6bd0c7-1900-429e-aa72-55bacb5f12b2-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361927    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v927v\" (UniqueName: \"kubernetes.io/projected/185df700-2f7c-4ff9-af4e-66a44b5bca2b-kube-api-access-v927v\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361942    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361955    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361967    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361981    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.361995    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362007    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362021    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362032    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185df700-2f7c-4ff9-af4e-66a44b5bca2b-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362044    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362238    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.362576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362428    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.363247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.362781    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:30.363247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.363049    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:30.363247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.363091    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.363247 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.363167    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:30.363563 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.363546    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:30.363754 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.363721    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.363871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.363848    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889" pod "a4d3ffe8-d8ed-4425-bf2d-824bb41af110" (UID: "a4d3ffe8-d8ed-4425-bf2d-824bb41af110"). InnerVolumeSpecName "kube-api-access-64889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.364371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.364357    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.364667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.364645    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c" pod "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" (UID: "c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232"). InnerVolumeSpecName "kube-api-access-w7t2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463600    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463643    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463657    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463669    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7t2c\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-kube-api-access-w7t2c\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463684    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463701    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463713    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463723    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463735    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64889\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-kube-api-access-64889\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463763    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.463811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.463774    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d3ffe8-d8ed-4425-bf2d-824bb41af110-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.868645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.868615    5967 generic.go:356] "Generic (PLEG): container finished" podID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerID="02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a" exitCode=0
Oct 30 20:47:30.869116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.868674    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"
Oct 30 20:47:30.869116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.868703    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" event={"ID":"185df700-2f7c-4ff9-af4e-66a44b5bca2b","Type":"ContainerDied","Data":"02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a"}
Oct 30 20:47:30.892764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.890854    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r" event={"ID":"185df700-2f7c-4ff9-af4e-66a44b5bca2b","Type":"ContainerDied","Data":"6fbe4bb00b6f66ea829f7881ff67d85cb4c53559d2dbb5ce2f2ff68708764bc2"}
Oct 30 20:47:30.892764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.890917    5967 scope.go:122] "RemoveContainer" containerID="02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a"
Oct 30 20:47:30.906318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.906119    5967 generic.go:356] "Generic (PLEG): container finished" podID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerID="20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f" exitCode=0
Oct 30 20:47:30.906318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.906303    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" event={"ID":"a4d3ffe8-d8ed-4425-bf2d-824bb41af110","Type":"ContainerDied","Data":"20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f"}
Oct 30 20:47:30.906435 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.906362    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt" event={"ID":"a4d3ffe8-d8ed-4425-bf2d-824bb41af110","Type":"ContainerDied","Data":"4b5ed928264847457f6bcaf531242a1f5f1c5c64c79bb3879b8121d59d4ebd41"}
Oct 30 20:47:30.906500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.906470    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"
Oct 30 20:47:30.917164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.916874    5967 generic.go:356] "Generic (PLEG): container finished" podID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerID="f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156" exitCode=0
Oct 30 20:47:30.917164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.917048    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"
Oct 30 20:47:30.917164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.917084    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" event={"ID":"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232","Type":"ContainerDied","Data":"f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156"}
Oct 30 20:47:30.917164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.917127    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr" event={"ID":"c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232","Type":"ContainerDied","Data":"c5417affdaa4e8dd2d6848df22f6bc0f34225a70ba26a1847adb5ea3dd390f05"}
Oct 30 20:47:30.920228 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.919893    5967 scope.go:122] "RemoveContainer" containerID="02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a"
Oct 30 20:47:30.921120 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:47:30.920809    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a\": container with ID starting with 02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a not found: ID does not exist" containerID="02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a"
Oct 30 20:47:30.921120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.920847    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a"} err="failed to get container status \"02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a\": rpc error: code = NotFound desc = could not find container \"02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a\": container with ID starting with 02138bc79f0b276001fea5b2659cae630381af002348e8c74925c09260efd76a not found: ID does not exist"
Oct 30 20:47:30.921120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.920869    5967 scope.go:122] "RemoveContainer" containerID="20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f"
Oct 30 20:47:30.922024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.921994    5967 generic.go:356] "Generic (PLEG): container finished" podID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerID="21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7" exitCode=0
Oct 30 20:47:30.922127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.922020    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" event={"ID":"4c6bd0c7-1900-429e-aa72-55bacb5f12b2","Type":"ContainerDied","Data":"21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7"}
Oct 30 20:47:30.922127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.922056    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"
Oct 30 20:47:30.922127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.922116    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n" event={"ID":"4c6bd0c7-1900-429e-aa72-55bacb5f12b2","Type":"ContainerDied","Data":"85685ce4bf3c26b4a8f5bb6065d5ccadbe147fb13d9d21cdbf6e855f4291967c"}
Oct 30 20:47:30.933101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.933087    5967 scope.go:122] "RemoveContainer" containerID="20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f"
Oct 30 20:47:30.933372 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:47:30.933348    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f\": container with ID starting with 20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f not found: ID does not exist" containerID="20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f"
Oct 30 20:47:30.933432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.933381    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f"} err="failed to get container status \"20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f\": rpc error: code = NotFound desc = could not find container \"20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f\": container with ID starting with 20a10fa85201e115c62a0080352ee93e54ceeb5df702755759f092a0e6213a4f not found: ID does not exist"
Oct 30 20:47:30.933432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.933398    5967 scope.go:122] "RemoveContainer" containerID="f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156"
Oct 30 20:47:30.944941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.944924    5967 scope.go:122] "RemoveContainer" containerID="f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156"
Oct 30 20:47:30.945235 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:47:30.945218    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156\": container with ID starting with f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156 not found: ID does not exist" containerID="f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156"
Oct 30 20:47:30.945288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.945244    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156"} err="failed to get container status \"f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156\": rpc error: code = NotFound desc = could not find container \"f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156\": container with ID starting with f0b536445aefb2fcaeedfa64a7665b4cd02c9dfc6d431dcb0caa0ffbdc224156 not found: ID does not exist"
Oct 30 20:47:30.945288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.945263    5967 scope.go:122] "RemoveContainer" containerID="21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7"
Oct 30 20:47:30.957592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.957569    5967 scope.go:122] "RemoveContainer" containerID="21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7"
Oct 30 20:47:30.958063 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:47:30.958037    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7\": container with ID starting with 21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7 not found: ID does not exist" containerID="21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7"
Oct 30 20:47:30.958184 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:30.958072    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7"} err="failed to get container status \"21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7\": rpc error: code = NotFound desc = could not find container \"21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7\": container with ID starting with 21a35453fc6a20d97b9e0821619ac3df2dbda90c0b1c100b5dd6e8485409aca7 not found: ID does not exist"
Oct 30 20:47:31.156943 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.156853    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"]
Oct 30 20:47:31.159939 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.159917    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r"]
Oct 30 20:47:31.321039 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.320979    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"]
Oct 30 20:47:31.325496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.325469    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt"]
Oct 30 20:47:31.440898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.440809    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"]
Oct 30 20:47:31.444222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.444192    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n"]
Oct 30 20:47:31.544238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.544206    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" path="/var/lib/kubelet/pods/185df700-2f7c-4ff9-af4e-66a44b5bca2b/volumes"
Oct 30 20:47:31.544689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.544675    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" path="/var/lib/kubelet/pods/4c6bd0c7-1900-429e-aa72-55bacb5f12b2/volumes"
Oct 30 20:47:31.545177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.545162    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" path="/var/lib/kubelet/pods/a4d3ffe8-d8ed-4425-bf2d-824bb41af110/volumes"
Oct 30 20:47:31.556001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.555974    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"]
Oct 30 20:47:31.562121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:31.562097    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr"]
Oct 30 20:47:32.001798 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:47:32.001761    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:47:33.556371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:33.556327    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" path="/var/lib/kubelet/pods/c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232/volumes"
Oct 30 20:47:40.008468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:40.008438    5967 generic.go:356] "Generic (PLEG): container finished" podID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerID="14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c" exitCode=0
Oct 30 20:47:40.008909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:40.008506    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" event={"ID":"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d","Type":"ContainerDied","Data":"14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"}
Oct 30 20:47:40.008909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:40.008896    5967 scope.go:122] "RemoveContainer" containerID="14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"
Oct 30 20:47:40.010364 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:40.010327    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:40.010560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:40.010418    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:40.010881 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:40.010839    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:41.038315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:41.038278    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" event={"ID":"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d","Type":"ContainerStarted","Data":"a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f"}
Oct 30 20:47:41.788764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:41.785125    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:47:41.788764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:41.785236    5967 patch_prober.go:29] interesting pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 20:47:41.788764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:41.785276    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 20:47:42.793954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:42.793917    5967 patch_prober.go:29] interesting pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 20:47:42.794355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:42.793972    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 20:47:43.793884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:43.793849    5967 patch_prober.go:29] interesting pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 20:47:43.794072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:43.793912    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 20:47:44.559484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.559448    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"]
Oct 30 20:47:44.560401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560375    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy"
Oct 30 20:47:44.560401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560399    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560412    5967 cpu_manager.go:414] "containerMap: removing container" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560419    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560436    5967 cpu_manager.go:414] "containerMap: removing container" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560441    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560458    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0bc425ce-c28a-43e1-93a5-e4263f17f903" containerName="revoke-keys"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560463    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc425ce-c28a-43e1-93a5-e4263f17f903" containerName="revoke-keys"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560470    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560475    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560588    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc425ce-c28a-43e1-93a5-e4263f17f903" containerName="revoke-keys"
Oct 30 20:47:44.560595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560598    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6bd0c7-1900-429e-aa72-55bacb5f12b2" containerName="istio-proxy"
Oct 30 20:47:44.560942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560613    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d3ffe8-d8ed-4425-bf2d-824bb41af110" containerName="istio-proxy"
Oct 30 20:47:44.560942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560619    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="185df700-2f7c-4ff9-af4e-66a44b5bca2b" containerName="istio-proxy"
Oct 30 20:47:44.560942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.560626    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a10ecd-1f2d-46c6-a4b7-b9aebc7d4232" containerName="istio-proxy"
Oct 30 20:47:44.563101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.563086    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564059    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564209 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564157    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564290 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564274    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564327 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564305    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564458 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564440    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564494 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564472    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564495    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564607    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvbb\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.564678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.564663    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.635578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.635553    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-061bda-openshift-default-dockercfg-j987r\""
Oct 30 20:47:44.666205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666175    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666241    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666281    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666297    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666360    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666381    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666402    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666457    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvbb\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666510    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666679    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666838 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666794    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666878    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.666942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.666889    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.667452 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.667428    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.697786 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.693182    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.706344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.706315    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.708456 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.708431    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"]
Oct 30 20:47:44.723333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.723309    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.723432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.723379    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvbb\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb\") pod \"e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") " pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:44.794155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.794120    5967 patch_prober.go:29] interesting pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 20:47:44.794805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.794770    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 20:47:44.893122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:44.893035    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:45.399732 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:45.399702    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"]
Oct 30 20:47:45.400073 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:47:45.400046    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2343b12_8707_454e_bdce_61f7ba30e16e.slice/crio-a080fb7e4563e7d9ec70cb600ded1e5f9db4af88908e0e020e09edab18ad8afb WatchSource:0}: Error finding container a080fb7e4563e7d9ec70cb600ded1e5f9db4af88908e0e020e09edab18ad8afb: Status 404 returned error can't find the container with id a080fb7e4563e7d9ec70cb600ded1e5f9db4af88908e0e020e09edab18ad8afb
Oct 30 20:47:45.402658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:45.402622    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:45.402792 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:45.402697    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:45.402874 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:45.402793    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:45.782086 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:45.782052    5967 patch_prober.go:29] interesting pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 20:47:45.782352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:45.782324    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 20:47:46.090904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.090824    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" event={"ID":"d2343b12-8707-454e-bdce-61f7ba30e16e","Type":"ContainerStarted","Data":"8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"}
Oct 30 20:47:46.091266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.090912    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" event={"ID":"d2343b12-8707-454e-bdce-61f7ba30e16e","Type":"ContainerStarted","Data":"a080fb7e4563e7d9ec70cb600ded1e5f9db4af88908e0e020e09edab18ad8afb"}
Oct 30 20:47:46.217328 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.217281    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podStartSLOduration=2.217267991 podStartE2EDuration="2.217267991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:46.214927564 +0000 UTC m=+1646.815729523" watchObservedRunningTime="2026-10-30 20:47:46.217267991 +0000 UTC m=+1646.818069997"
Oct 30 20:47:46.785875 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.785846    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:47:46.786196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.786176    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:47:46.786798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.786776    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:47:46.901307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.901270    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:46.908035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:46.908002    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:47.099336 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:47.099251    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:47.100065 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:47.100043    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:47:51.442067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.442036    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"]
Oct 30 20:47:51.445537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.445510    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.545997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.545968    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"]
Oct 30 20:47:51.551270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.551240    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-034eba-openshift-default-dockercfg-xg47p\""
Oct 30 20:47:51.597758 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.597704    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598092 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598070    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598307    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598486 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598470    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598600    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598773    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598812    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.598937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598865    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.599052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.598956    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701223    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701276    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701323    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701446    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701500 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701485    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701528    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701548    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701601 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701592    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701689    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701721    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701851    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.701902 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701880    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.702020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.701995    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.702263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.702246    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.704079 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.704058    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.704329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.704311    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.759830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.759796    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:51.760058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:51.759950    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token\") pod \"e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") " pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:52.059285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:52.059249    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:52.539004 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:52.534867    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"]
Oct 30 20:47:52.574198 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:52.574075    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:52.574198 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:52.574170    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:52.574415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:52.574387    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:53.155374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:53.155339    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" event={"ID":"9dbc28a0-6c60-40a2-ac35-be0f154aea4b","Type":"ContainerStarted","Data":"d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"}
Oct 30 20:47:53.155661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:53.155404    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" event={"ID":"9dbc28a0-6c60-40a2-ac35-be0f154aea4b","Type":"ContainerStarted","Data":"2d5018da9a00862549828ee5ce2a17c12d6d283f09b65cdff6051f454d7abbba"}
Oct 30 20:47:54.059747 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:54.059719    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:54.065454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:54.065433    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:54.165860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:54.164705    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:54.165860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:54.165809    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:47:54.233734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:47:54.233684    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podStartSLOduration=3.233666654 podStartE2EDuration="3.233666654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:53.371767215 +0000 UTC m=+1653.972569174" watchObservedRunningTime="2026-10-30 20:47:54.233666654 +0000 UTC m=+1654.834468614"
Oct 30 20:48:01.980559 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:48:01.980516    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:48:10.126951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:10.126919    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"]
Oct 30 20:48:10.127468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:10.127107    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy" containerID="cri-o://9a4a1670eef0e03637c663132f2180221f156d59adf3627a16ec92fedb217a24" gracePeriod=30
Oct 30 20:48:10.176093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:10.176067    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"]
Oct 30 20:48:10.176289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:10.176254    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy" containerID="cri-o://7541845f5641ea7a9980cc7c5c90375c589f57e0f79136ed89a63a6a9c0d107f" gracePeriod=30
Oct 30 20:48:15.328643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.328593    5967 generic.go:356] "Generic (PLEG): container finished" podID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerID="9a4a1670eef0e03637c663132f2180221f156d59adf3627a16ec92fedb217a24" exitCode=0
Oct 30 20:48:15.329145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.328873    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" event={"ID":"75348b55-bae3-4fc5-9e4d-cccef1a7e338","Type":"ContainerDied","Data":"9a4a1670eef0e03637c663132f2180221f156d59adf3627a16ec92fedb217a24"}
Oct 30 20:48:15.331515 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.331484    5967 generic.go:356] "Generic (PLEG): container finished" podID="97a87637-d9da-44bc-af84-44750f82613b" containerID="7541845f5641ea7a9980cc7c5c90375c589f57e0f79136ed89a63a6a9c0d107f" exitCode=0
Oct 30 20:48:15.331644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.331614    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" event={"ID":"97a87637-d9da-44bc-af84-44750f82613b","Type":"ContainerDied","Data":"7541845f5641ea7a9980cc7c5c90375c589f57e0f79136ed89a63a6a9c0d107f"}
Oct 30 20:48:15.482508 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.482485    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:48:15.486228 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.486207    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:48:15.584635 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584604    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.584836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584672    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.584836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584695    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.584836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584717    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.584836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584808    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584851    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584907    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584930    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.584996    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585039    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585042    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.585102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585081    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585157    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585194    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585245    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585245    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585323    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585398    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585424    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy\") pod \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\" (UID: \"75348b55-bae3-4fc5-9e4d-cccef1a7e338\") "
Oct 30 20:48:15.585455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585432    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.585924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585465    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert\") pod \"97a87637-d9da-44bc-af84-44750f82613b\" (UID: \"97a87637-d9da-44bc-af84-44750f82613b\") "
Oct 30 20:48:15.585924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.585769    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:15.586107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586085    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.586436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586409    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.586538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586444    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.586538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586460    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.586538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586476    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.586538 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586490    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.586801 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.586711    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.587154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.587021    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.587393 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.587368    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:15.587859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.587825    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.588188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.588158    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:48:15.588344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.588323    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.588664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.588636    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.590913 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.590888    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:48:15.592084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.592058    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.592167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.592149    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.592210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.592177    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "kube-api-access-hxz7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.592210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.592202    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token" pod "97a87637-d9da-44bc-af84-44750f82613b" (UID: "97a87637-d9da-44bc-af84-44750f82613b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.593614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.593589    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr" pod "75348b55-bae3-4fc5-9e4d-cccef1a7e338" (UID: "75348b55-bae3-4fc5-9e4d-cccef1a7e338"). InnerVolumeSpecName "kube-api-access-8d7tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687409    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687437    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687448    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7tr\" (UniqueName: \"kubernetes.io/projected/75348b55-bae3-4fc5-9e4d-cccef1a7e338-kube-api-access-8d7tr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687456    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687465    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687474    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a87637-d9da-44bc-af84-44750f82613b-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687481    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687489    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75348b55-bae3-4fc5-9e4d-cccef1a7e338-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.687495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687497    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxz7d\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-kube-api-access-hxz7d\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.688136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687506    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a87637-d9da-44bc-af84-44750f82613b-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.688136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687514    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75348b55-bae3-4fc5-9e4d-cccef1a7e338-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.688136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687523    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a87637-d9da-44bc-af84-44750f82613b-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.688136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:15.687531    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a87637-d9da-44bc-af84-44750f82613b-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:16.340597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.340561    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"
Oct 30 20:48:16.341051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.340595    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s" event={"ID":"75348b55-bae3-4fc5-9e4d-cccef1a7e338","Type":"ContainerDied","Data":"c402373e6878d2b47a7b182075e67bf20a9f4bfdb40e9c796c7f6ec6fe5dcfe3"}
Oct 30 20:48:16.341051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.340665    5967 scope.go:122] "RemoveContainer" containerID="9a4a1670eef0e03637c663132f2180221f156d59adf3627a16ec92fedb217a24"
Oct 30 20:48:16.342519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.342497    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"
Oct 30 20:48:16.342630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.342502    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff" event={"ID":"97a87637-d9da-44bc-af84-44750f82613b","Type":"ContainerDied","Data":"bec9caefbade56c5700c57bdb58fe1798511b2767b7c3bfccd53bbfe6ec73d5f"}
Oct 30 20:48:16.352806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.352790    5967 scope.go:122] "RemoveContainer" containerID="7541845f5641ea7a9980cc7c5c90375c589f57e0f79136ed89a63a6a9c0d107f"
Oct 30 20:48:16.518150 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.518120    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"]
Oct 30 20:48:16.568227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.568191    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff"]
Oct 30 20:48:16.678329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.678294    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"]
Oct 30 20:48:16.683620 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:16.683595    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s"]
Oct 30 20:48:17.548472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:17.548432    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" path="/var/lib/kubelet/pods/75348b55-bae3-4fc5-9e4d-cccef1a7e338/volumes"
Oct 30 20:48:17.549149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:17.549130    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a87637-d9da-44bc-af84-44750f82613b" path="/var/lib/kubelet/pods/97a87637-d9da-44bc-af84-44750f82613b/volumes"
Oct 30 20:48:21.132959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.132931    5967 scope.go:122] "RemoveContainer" containerID="d53c940d36d64712f52251290c19db6555b64a10044922545d034965711127dd"
Oct 30 20:48:21.145574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.145554    5967 scope.go:122] "RemoveContainer" containerID="78ff5f17cff6605bbabaf0a6d972862da5aa7fd87e4b31f131acd54b5f594391"
Oct 30 20:48:21.157679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.157504    5967 scope.go:122] "RemoveContainer" containerID="a39ec9185fe324a08f3ef1bf6e59b797e547d68726b4dcf0f9bbdf4adf75d4ce"
Oct 30 20:48:21.169388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.169368    5967 scope.go:122] "RemoveContainer" containerID="74308487e81ee40e3c29d55508eecc130804ce664af24822e2f571cc560f14b0"
Oct 30 20:48:21.178689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.178665    5967 scope.go:122] "RemoveContainer" containerID="f8039721d61ac6687727316022b2b37b51da2b339ee5a3762432bb4ab539bce2"
Oct 30 20:48:21.187695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.187673    5967 scope.go:122] "RemoveContainer" containerID="ec22e27721d07b799a36a1472cef80bf58a188fd277ba789863df25ef3736ede"
Oct 30 20:48:21.196396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:21.196381    5967 scope.go:122] "RemoveContainer" containerID="0a003be3606f4eb47b7db2a27fa47e1ae51d0655476cc5ee4686caae2c249cec"
Oct 30 20:48:30.001275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.001242    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"]
Oct 30 20:48:30.002052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.002032    5967 cpu_manager.go:414] "containerMap: removing container" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy"
Oct 30 20:48:30.002052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.002052    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy"
Oct 30 20:48:30.002200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.002096    5967 cpu_manager.go:414] "containerMap: removing container" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy"
Oct 30 20:48:30.002200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.002101    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy"
Oct 30 20:48:30.002200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.002194    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="75348b55-bae3-4fc5-9e4d-cccef1a7e338" containerName="istio-proxy"
Oct 30 20:48:30.002309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.002207    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a87637-d9da-44bc-af84-44750f82613b" containerName="istio-proxy"
Oct 30 20:48:30.007798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.007781    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.063828 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.063803    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"]
Oct 30 20:48:30.105477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.105450    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4ht\" (UniqueName: \"kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.105642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.105618    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.105751 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.105667    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.107830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.107803    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:48:30.107830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.107821    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:48:30.107947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.107850    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:48:30.109446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.109431    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:48:30.206551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.206524    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.206688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.206558    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.206688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.206682    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4ht\" (UniqueName: \"kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.207196 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.207171    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.208989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.208961    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.263454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.263406    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4ht\" (UniqueName: \"kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht\") pod \"maas-api-revoke-sub-33007785-vhz4h\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.322017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.321997    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:30.702292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:30.702252    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"]
Oct 30 20:48:30.706208 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:48:30.706176    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04824aa9_8a45_4ddb_aafd_b0a186e28062.slice/crio-77da42340ca2ef628487dcfba8eacde467751562a3698b6110daaec54abe1e75 WatchSource:0}: Error finding container 77da42340ca2ef628487dcfba8eacde467751562a3698b6110daaec54abe1e75: Status 404 returned error can't find the container with id 77da42340ca2ef628487dcfba8eacde467751562a3698b6110daaec54abe1e75
Oct 30 20:48:31.463480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:31.463392    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h" event={"ID":"04824aa9-8a45-4ddb-aafd-b0a186e28062","Type":"ContainerStarted","Data":"bb95572cab973b3907738478db6c35b631f7af9779a29a27d3f5c0f4ec134222"}
Oct 30 20:48:31.463480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:31.463480    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h" event={"ID":"04824aa9-8a45-4ddb-aafd-b0a186e28062","Type":"ContainerStarted","Data":"77da42340ca2ef628487dcfba8eacde467751562a3698b6110daaec54abe1e75"}
Oct 30 20:48:31.582037 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:31.581991    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h" podStartSLOduration=2.581976502 podStartE2EDuration="2.581976502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:31.578429301 +0000 UTC m=+1692.179231271" watchObservedRunningTime="2026-10-30 20:48:31.581976502 +0000 UTC m=+1692.182778462"
Oct 30 20:48:31.980948 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:48:31.980914    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:48:32.471409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:32.471374    5967 generic.go:356] "Generic (PLEG): container finished" podID="04824aa9-8a45-4ddb-aafd-b0a186e28062" containerID="bb95572cab973b3907738478db6c35b631f7af9779a29a27d3f5c0f4ec134222" exitCode=0
Oct 30 20:48:32.471842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:32.471465    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h" event={"ID":"04824aa9-8a45-4ddb-aafd-b0a186e28062","Type":"ContainerDied","Data":"bb95572cab973b3907738478db6c35b631f7af9779a29a27d3f5c0f4ec134222"}
Oct 30 20:48:33.636446 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.636424    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:33.754661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.754591    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca\") pod \"04824aa9-8a45-4ddb-aafd-b0a186e28062\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") "
Oct 30 20:48:33.754851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.754664    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca\") pod \"04824aa9-8a45-4ddb-aafd-b0a186e28062\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") "
Oct 30 20:48:33.754911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.754862    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht\" (UniqueName: \"kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht\") pod \"04824aa9-8a45-4ddb-aafd-b0a186e28062\" (UID: \"04824aa9-8a45-4ddb-aafd-b0a186e28062\") "
Oct 30 20:48:33.755313 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.755288    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca" pod "04824aa9-8a45-4ddb-aafd-b0a186e28062" (UID: "04824aa9-8a45-4ddb-aafd-b0a186e28062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:33.757417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.757395    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca" pod "04824aa9-8a45-4ddb-aafd-b0a186e28062" (UID: "04824aa9-8a45-4ddb-aafd-b0a186e28062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:33.758220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.758200    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht" pod "04824aa9-8a45-4ddb-aafd-b0a186e28062" (UID: "04824aa9-8a45-4ddb-aafd-b0a186e28062"). InnerVolumeSpecName "kube-api-access-dh4ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:33.857049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.857020    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:33.857049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.857047    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04824aa9-8a45-4ddb-aafd-b0a186e28062-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:33.857228 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:33.857058    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4ht\" (UniqueName: \"kubernetes.io/projected/04824aa9-8a45-4ddb-aafd-b0a186e28062-kube-api-access-dh4ht\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:34.487098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:34.487071    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"
Oct 30 20:48:34.487278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:34.487100    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h" event={"ID":"04824aa9-8a45-4ddb-aafd-b0a186e28062","Type":"ContainerDied","Data":"77da42340ca2ef628487dcfba8eacde467751562a3698b6110daaec54abe1e75"}
Oct 30 20:48:34.487278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:34.487150    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77da42340ca2ef628487dcfba8eacde467751562a3698b6110daaec54abe1e75"
Oct 30 20:48:56.049175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:56.049143    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"]
Oct 30 20:48:56.098926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:56.098895    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c08e0c4-7thrx"]
Oct 30 20:48:57.543648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:48:57.543615    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044de5a6-9abd-4ea5-89ff-720297490928" path="/var/lib/kubelet/pods/044de5a6-9abd-4ea5-89ff-720297490928/volumes"
Oct 30 20:49:01.980535 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:49:01.980448    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:49:07.189975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.189947    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"]
Oct 30 20:49:07.190631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.190614    5967 cpu_manager.go:414] "containerMap: removing container" podUID="04824aa9-8a45-4ddb-aafd-b0a186e28062" containerName="revoke-keys"
Oct 30 20:49:07.190704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.190632    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04824aa9-8a45-4ddb-aafd-b0a186e28062" containerName="revoke-keys"
Oct 30 20:49:07.190787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.190776    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="04824aa9-8a45-4ddb-aafd-b0a186e28062" containerName="revoke-keys"
Oct 30 20:49:07.194560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.194540    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.251513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.251488    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"]
Oct 30 20:49:07.255206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.255187    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.255317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.255230    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skndr\" (UniqueName: \"kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.255415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.255403    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.296797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.296775    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:07.296797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.296787    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:49:07.296989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.296842    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:49:07.296989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.296931    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:49:07.356358 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.356330    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.356521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.356494    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.356588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.356554    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skndr\" (UniqueName: \"kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.357066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.357043    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.358820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.358802    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.413288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.413260    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skndr\" (UniqueName: \"kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr\") pod \"maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.508852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.508822    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:07.887818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:07.887785    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"]
Oct 30 20:49:07.889367 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:49:07.889335    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5247db_5bb8_4bf0_b951_2284a1a25164.slice/crio-4dde8e9d2046deb68773a79336bd985c78294598e9fc50ff5c625a6e94be6e62 WatchSource:0}: Error finding container 4dde8e9d2046deb68773a79336bd985c78294598e9fc50ff5c625a6e94be6e62: Status 404 returned error can't find the container with id 4dde8e9d2046deb68773a79336bd985c78294598e9fc50ff5c625a6e94be6e62
Oct 30 20:49:08.744904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:08.744803    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4" event={"ID":"8c5247db-5bb8-4bf0-b951-2284a1a25164","Type":"ContainerStarted","Data":"619387df1f960bf34383665414f1a33e29656537b08fd38c018e864d34cbee91"}
Oct 30 20:49:08.744904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:08.744865    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4" event={"ID":"8c5247db-5bb8-4bf0-b951-2284a1a25164","Type":"ContainerStarted","Data":"4dde8e9d2046deb68773a79336bd985c78294598e9fc50ff5c625a6e94be6e62"}
Oct 30 20:49:09.752886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.752851    5967 generic.go:356] "Generic (PLEG): container finished" podID="8c5247db-5bb8-4bf0-b951-2284a1a25164" containerID="619387df1f960bf34383665414f1a33e29656537b08fd38c018e864d34cbee91" exitCode=0
Oct 30 20:49:09.753317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.752937    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4" event={"ID":"8c5247db-5bb8-4bf0-b951-2284a1a25164","Type":"ContainerDied","Data":"619387df1f960bf34383665414f1a33e29656537b08fd38c018e864d34cbee91"}
Oct 30 20:49:09.914463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.914442    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:09.992322 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.992302    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca\") pod \"8c5247db-5bb8-4bf0-b951-2284a1a25164\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") "
Oct 30 20:49:09.992428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.992361    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr\" (UniqueName: \"kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr\") pod \"8c5247db-5bb8-4bf0-b951-2284a1a25164\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") "
Oct 30 20:49:09.992428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.992390    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca\") pod \"8c5247db-5bb8-4bf0-b951-2284a1a25164\" (UID: \"8c5247db-5bb8-4bf0-b951-2284a1a25164\") "
Oct 30 20:49:09.992630 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.992610    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca" pod "8c5247db-5bb8-4bf0-b951-2284a1a25164" (UID: "8c5247db-5bb8-4bf0-b951-2284a1a25164"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:09.993006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.992955    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:09.994501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.994482    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr" pod "8c5247db-5bb8-4bf0-b951-2284a1a25164" (UID: "8c5247db-5bb8-4bf0-b951-2284a1a25164"). InnerVolumeSpecName "kube-api-access-skndr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:09.994644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:09.994625    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca" pod "8c5247db-5bb8-4bf0-b951-2284a1a25164" (UID: "8c5247db-5bb8-4bf0-b951-2284a1a25164"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:10.094894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:10.094863    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skndr\" (UniqueName: \"kubernetes.io/projected/8c5247db-5bb8-4bf0-b951-2284a1a25164-kube-api-access-skndr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:10.094894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:10.094891    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5247db-5bb8-4bf0-b951-2284a1a25164-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:10.761217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:10.761186    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"
Oct 30 20:49:10.761217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:10.761199    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4" event={"ID":"8c5247db-5bb8-4bf0-b951-2284a1a25164","Type":"ContainerDied","Data":"4dde8e9d2046deb68773a79336bd985c78294598e9fc50ff5c625a6e94be6e62"}
Oct 30 20:49:10.761658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:10.761239    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dde8e9d2046deb68773a79336bd985c78294598e9fc50ff5c625a6e94be6e62"
Oct 30 20:49:18.495905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:18.495847    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:18.708602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:18.708563    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 20:49:18.821357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:18.821261    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:18.892718 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:18.891696    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:18.964765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:18.954819    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.201026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.200854    5967 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.257164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.257053    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.326020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.325972    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.417211 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.415069    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.546770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.505813    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.546770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.506241    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:49:19.711841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.711802    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:19.751521 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.751494    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:49:19.752295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.752279    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:49:19.816305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:19.815290    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 20:49:26.017564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.017534    5967 generic.go:356] "Generic (PLEG): container finished" podID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerID="8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d" exitCode=0
Oct 30 20:49:26.018155 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.017613    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" event={"ID":"d2343b12-8707-454e-bdce-61f7ba30e16e","Type":"ContainerDied","Data":"8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"}
Oct 30 20:49:26.018227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.018200    5967 scope.go:122] "RemoveContainer" containerID="8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"
Oct 30 20:49:26.020054 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.020027    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:26.020151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.020098    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:26.020151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.020132    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:26.680466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.680432    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"]
Oct 30 20:49:26.681122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.681106    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8c5247db-5bb8-4bf0-b951-2284a1a25164" containerName="revoke-keys"
Oct 30 20:49:26.681122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.681122    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5247db-5bb8-4bf0-b951-2284a1a25164" containerName="revoke-keys"
Oct 30 20:49:26.681258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.681249    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5247db-5bb8-4bf0-b951-2284a1a25164" containerName="revoke-keys"
Oct 30 20:49:26.684988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.684967    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.737762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.737268    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-ab9c38-openshift-default-dockercfg-hb5r7\""
Oct 30 20:49:26.762519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762493    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.762662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762565    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.762662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762653    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.762804 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762790    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.762846 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762835    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.762942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762927    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.762981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762955    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.763014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.762996    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsn67\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.763058 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.763019    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.790441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.790411    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"]
Oct 30 20:49:26.863907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.863867    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsn67\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.863924    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864019    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864051    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864146    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864267    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864344    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864401    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864415    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864471    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864625    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864725 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864701    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.864933 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.864847    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.865204 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.865183    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.866818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.866795    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.867160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.867133    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.921819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.921781    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsn67\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:26.922199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:26.922172    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token\") pod \"e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") " pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:27.008780 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.005089    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:27.142092 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.142058    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" event={"ID":"d2343b12-8707-454e-bdce-61f7ba30e16e","Type":"ContainerStarted","Data":"7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011"}
Oct 30 20:49:27.580035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.579989    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"]
Oct 30 20:49:27.591420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.589857    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:27.591420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.589949    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:27.591420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.589991    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:27.901161 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.901065    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:49:27.904020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.903991    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:49:27.904182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:27.904043    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 20:49:28.153347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:28.153259    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" event={"ID":"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031","Type":"ContainerStarted","Data":"a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"}
Oct 30 20:49:28.153347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:28.153319    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" event={"ID":"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031","Type":"ContainerStarted","Data":"c65f5ad2adafb5b94d1b6f117307db6781e8b77ab38dcd659d3e5a48e4406e79"}
Oct 30 20:49:28.902055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:28.902022    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:49:28.902232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:28.902072    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 20:49:29.006048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.006009    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:29.011937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.011903    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:29.135066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.135015    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podStartSLOduration=3.135000817 podStartE2EDuration="3.135000817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:28.292017458 +0000 UTC m=+1748.892819433" watchObservedRunningTime="2026-10-30 20:49:29.135000817 +0000 UTC m=+1749.735802777"
Oct 30 20:49:29.159928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.159854    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:29.160721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.160701    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:49:29.901916 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.901875    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:49:29.902098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:29.901955    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 20:49:30.902401 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:30.902372    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:49:30.902846 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:30.902427    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 20:49:31.254393 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.254352    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"]
Oct 30 20:49:31.258756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.258718    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.259790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.259723    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfcq\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.259932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.259853    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260047 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260023    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260199    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260244    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260286    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260383    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260467    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.260537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.260516    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.314764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.313780    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f8ac60-openshift-default-dockercfg-nzftc\""
Oct 30 20:49:31.383764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.383666    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.383961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.383810    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.444766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.425795    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.526850    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.526920    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.526968    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.527065    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.527123    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.527196    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfcq\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.527282    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.527799    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.530941 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.528064    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.646789 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.626978    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.673222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.673182    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.800886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.775734    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.800886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.777136    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"]
Oct 30 20:49:31.800886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.780567    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfcq\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.800886 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.789440    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.824769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.802015    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy\") pod \"e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") " pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.894103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.894054    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:31.901745 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.901714    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:49:31.901864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:31.901787    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 20:49:32.017115 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:49:32.015946    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:49:32.110863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.105894    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:49:32.359067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.359035    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"]
Oct 30 20:49:32.359067 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:49:32.359047    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6926a1c_820c_48b6_b3fd_31b4026a7804.slice/crio-e02d1ec4f21434677e26ae4f9317a489895a33ec69582c966e861531debec223 WatchSource:0}: Error finding container e02d1ec4f21434677e26ae4f9317a489895a33ec69582c966e861531debec223: Status 404 returned error can't find the container with id e02d1ec4f21434677e26ae4f9317a489895a33ec69582c966e861531debec223
Oct 30 20:49:32.362536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.362500    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:32.362632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.362586    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:32.362672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.362630    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:49:32.906284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.906258    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:49:32.907420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:32.907398    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:49:33.216085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:33.216055    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" event={"ID":"b6926a1c-820c-48b6-b3fd-31b4026a7804","Type":"ContainerStarted","Data":"6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"}
Oct 30 20:49:33.216557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:33.216116    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" event={"ID":"b6926a1c-820c-48b6-b3fd-31b4026a7804","Type":"ContainerStarted","Data":"e02d1ec4f21434677e26ae4f9317a489895a33ec69582c966e861531debec223"}
Oct 30 20:49:33.335791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:33.335714    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podStartSLOduration=2.335700725 podStartE2EDuration="2.335700725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:33.334718201 +0000 UTC m=+1753.935520194" watchObservedRunningTime="2026-10-30 20:49:33.335700725 +0000 UTC m=+1753.936502684"
Oct 30 20:49:33.895192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:33.895157    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:33.900600 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:33.900576    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:34.223323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:34.223298    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:34.224337 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:34.224317    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:49:46.069178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:46.069140    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"]
Oct 30 20:49:46.118433 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:46.118404    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806d51b0-ztvsg"]
Oct 30 20:49:47.558785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:47.558009    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4fdbbc-f573-41a6-8b16-91213479784f" path="/var/lib/kubelet/pods/6f4fdbbc-f573-41a6-8b16-91213479784f/volumes"
Oct 30 20:49:49.711144 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:49.711109    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"]
Oct 30 20:49:49.760853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:49.760826    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c650f2b7-7txs4"]
Oct 30 20:49:51.544093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:51.544059    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5247db-5bb8-4bf0-b951-2284a1a25164" path="/var/lib/kubelet/pods/8c5247db-5bb8-4bf0-b951-2284a1a25164/volumes"
Oct 30 20:49:57.335326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:57.335292    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"]
Oct 30 20:49:57.335793 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:57.335519    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy" containerID="cri-o://88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff" gracePeriod=30
Oct 30 20:49:58.894766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:58.891818    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"]
Oct 30 20:49:58.898606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:58.898580    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:58.951541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:58.951518    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c3d13a-openshift-default-dockercfg-ktbqz\""
Oct 30 20:49:59.007130 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.007095    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"]
Oct 30 20:49:59.015202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015163    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015264    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015410    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015436    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015504    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015558    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015591    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015671    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.015839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.015790    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfstq\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.132774 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.125697    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.133351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.133322    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.158934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.158841    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.193961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.193924    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.194223    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.194273    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.194379    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.194492    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfstq\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.194571    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.195445    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.144076    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.197763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.127196    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.228764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.199591    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.229063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.201541    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.259697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.259622    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfstq\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.328018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.326102    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.328499 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.328429    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.329235 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.329213    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token\") pod \"e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") " pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.516930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.516900    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:49:59.999202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:49:59.999176    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"]
Oct 30 20:49:59.999897 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:49:59.999872    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c1c1e0_90bb_45fa_b30b_d9d235d502b2.slice/crio-45d4b9ac72c0ad020deda019ee543daa95ab1fa3a98505775c1853ab5dd2d26e WatchSource:0}: Error finding container 45d4b9ac72c0ad020deda019ee543daa95ab1fa3a98505775c1853ab5dd2d26e: Status 404 returned error can't find the container with id 45d4b9ac72c0ad020deda019ee543daa95ab1fa3a98505775c1853ab5dd2d26e
Oct 30 20:50:00.002556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:00.002516    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:00.002662 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:00.002598    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:00.002726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:00.002658    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:00.635271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:00.635237    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" event={"ID":"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2","Type":"ContainerStarted","Data":"b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"}
Oct 30 20:50:00.635447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:00.635317    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" event={"ID":"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2","Type":"ContainerStarted","Data":"45d4b9ac72c0ad020deda019ee543daa95ab1fa3a98505775c1853ab5dd2d26e"}
Oct 30 20:50:00.760975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:00.760916    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podStartSLOduration=2.7608984039999998 podStartE2EDuration="2.760898404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:00.759312039 +0000 UTC m=+1781.360113999" watchObservedRunningTime="2026-10-30 20:50:00.760898404 +0000 UTC m=+1781.361700365"
Oct 30 20:50:01.517504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:01.517467    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:50:01.523483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:01.523454    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:50:01.644084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:01.644058    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:50:01.645236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:01.645210    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:50:01.984714 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:50:01.984675    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:50:02.627889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.627863    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:50:02.652369 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.652338    5967 generic.go:356] "Generic (PLEG): container finished" podID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerID="88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff" exitCode=0
Oct 30 20:50:02.652490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.652385    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"
Oct 30 20:50:02.652490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.652427    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" event={"ID":"a4947cbd-7f16-417b-9730-8b226f74a4f7","Type":"ContainerDied","Data":"88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff"}
Oct 30 20:50:02.652490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.652482    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk" event={"ID":"a4947cbd-7f16-417b-9730-8b226f74a4f7","Type":"ContainerDied","Data":"67f318abdfed86a96457a986eca7f7c0b79fc2cc04a139b3c2c071deff8aa934"}
Oct 30 20:50:02.652674 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.652510    5967 scope.go:122] "RemoveContainer" containerID="88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff"
Oct 30 20:50:02.664066 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.664044    5967 scope.go:122] "RemoveContainer" containerID="b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"
Oct 30 20:50:02.675645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.675628    5967 scope.go:122] "RemoveContainer" containerID="88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff"
Oct 30 20:50:02.675929 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:50:02.675911    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff\": container with ID starting with 88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff not found: ID does not exist" containerID="88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff"
Oct 30 20:50:02.675996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.675935    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff"} err="failed to get container status \"88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff\": rpc error: code = NotFound desc = could not find container \"88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff\": container with ID starting with 88b08ca8a960a3fc3923f9dcacb535b0bdc807f2f663a27bac12ae6d9ad771ff not found: ID does not exist"
Oct 30 20:50:02.675996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.675950    5967 scope.go:122] "RemoveContainer" containerID="b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"
Oct 30 20:50:02.676211 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:50:02.676198    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4\": container with ID starting with b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4 not found: ID does not exist" containerID="b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"
Oct 30 20:50:02.676255 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.676215    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4"} err="failed to get container status \"b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4\": rpc error: code = NotFound desc = could not find container \"b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4\": container with ID starting with b77027832c825423a2773027fd0bbee8d6e0aa4074cbdc641da1ed8af717b8c4 not found: ID does not exist"
Oct 30 20:50:02.781850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.781824    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.781982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.781856    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.781982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.781877    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.781982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.781929    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.781982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.781952    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.782202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782056    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.782202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782151    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.782202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782187    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.782355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782252    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket\") pod \"a4947cbd-7f16-417b-9730-8b226f74a4f7\" (UID: \"a4947cbd-7f16-417b-9730-8b226f74a4f7\") "
Oct 30 20:50:02.782413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782368    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:02.782498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782470    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:02.782658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782638    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:02.782850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782823    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:02.782994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.782969    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:02.783319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.783284    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4947cbd-7f16-417b-9730-8b226f74a4f7-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.783319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.783310    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.783461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.783325    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.783461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.783339    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.783461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.783353    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.784588 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.784547    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "kube-api-access-nj7t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:02.784674 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.784650    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:02.784725 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.784662    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:02.784987 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.784967    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo" pod "a4947cbd-7f16-417b-9730-8b226f74a4f7" (UID: "a4947cbd-7f16-417b-9730-8b226f74a4f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:50:02.884619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.884547    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.884619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.884573    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj7t6\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-kube-api-access-nj7t6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.884619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.884584    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:02.884619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:02.884598    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4947cbd-7f16-417b-9730-8b226f74a4f7-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:03.074597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:03.074537    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"]
Oct 30 20:50:03.078800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:03.078775    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk"]
Oct 30 20:50:03.548514 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:03.547879    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" path="/var/lib/kubelet/pods/a4947cbd-7f16-417b-9730-8b226f74a4f7/volumes"
Oct 30 20:50:04.670799 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:04.670763    5967 generic.go:356] "Generic (PLEG): container finished" podID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerID="d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665" exitCode=0
Oct 30 20:50:04.670799 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:04.670767    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" event={"ID":"9dbc28a0-6c60-40a2-ac35-be0f154aea4b","Type":"ContainerDied","Data":"d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"}
Oct 30 20:50:04.671328 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:04.671141    5967 scope.go:122] "RemoveContainer" containerID="d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"
Oct 30 20:50:04.672669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:04.672641    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:04.672731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:04.672709    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:04.672808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:04.672783    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:05.699658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:05.699624    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" event={"ID":"9dbc28a0-6c60-40a2-ac35-be0f154aea4b","Type":"ContainerStarted","Data":"c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d"}
Oct 30 20:50:06.059691 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:06.059658    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:50:06.060997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:06.060971    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:50:06.061123 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:06.061022    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 20:50:07.059815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:07.059786    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:50:07.060211 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:07.059838    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 20:50:08.065464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:08.065430    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:50:08.067374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:08.067337    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 20:50:09.059928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:09.059896    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:50:09.060142 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:09.059963    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 20:50:09.164671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:09.164633    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:50:10.060501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:10.060460    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 20:50:10.060694 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:10.060532    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 20:50:11.068195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.068157    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:50:11.069711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.069690    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:50:11.545527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.545498    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"]
Oct 30 20:50:11.546110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.546096    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy"
Oct 30 20:50:11.546160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.546112    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy"
Oct 30 20:50:11.546252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.546243    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy"
Oct 30 20:50:11.546297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.546257    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy"
Oct 30 20:50:11.548655 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.548640    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.600949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.600922    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:50:11.601142 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.601122    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:50:11.601351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.601330    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:11.602304 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.601488    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:50:11.603880 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.603854    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"]
Oct 30 20:50:11.681611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.681576    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.681841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.681778    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.681950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.681933    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjl2\" (UniqueName: \"kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.783293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.783261    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjl2\" (UniqueName: \"kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.783439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.783418    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.783514 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.783498    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.784028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.784007    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.785932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.785911    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.841714 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.841664    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjl2\" (UniqueName: \"kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2\") pod \"maas-api-revoke-sub-389ffc2a-xnpt9\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:11.861264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:11.861245    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:12.240623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:12.240595    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"]
Oct 30 20:50:12.242272 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:50:12.242248    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3772bdaa_a3ac_441c_b46b_2dc0573937de.slice/crio-a0b7cfeebb0962de861c47bcdac24b261f293b06cc47c11329a7a892c6422b99 WatchSource:0}: Error finding container a0b7cfeebb0962de861c47bcdac24b261f293b06cc47c11329a7a892c6422b99: Status 404 returned error can't find the container with id a0b7cfeebb0962de861c47bcdac24b261f293b06cc47c11329a7a892c6422b99
Oct 30 20:50:12.774509 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:12.774432    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9" event={"ID":"3772bdaa-a3ac-441c-b46b-2dc0573937de","Type":"ContainerStarted","Data":"fed4a782f13e9be23dcdd4f32a657e6c814e85ae194e1d852c417f88a91baae0"}
Oct 30 20:50:12.774509 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:12.774485    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9" event={"ID":"3772bdaa-a3ac-441c-b46b-2dc0573937de","Type":"ContainerStarted","Data":"a0b7cfeebb0962de861c47bcdac24b261f293b06cc47c11329a7a892c6422b99"}
Oct 30 20:50:12.893159 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:12.893112    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9" podStartSLOduration=1.8930987099999999 podStartE2EDuration="1.89309871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:12.889535123 +0000 UTC m=+1793.490337083" watchObservedRunningTime="2026-10-30 20:50:12.89309871 +0000 UTC m=+1793.493900670"
Oct 30 20:50:13.783266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:13.783225    5967 generic.go:356] "Generic (PLEG): container finished" podID="3772bdaa-a3ac-441c-b46b-2dc0573937de" containerID="fed4a782f13e9be23dcdd4f32a657e6c814e85ae194e1d852c417f88a91baae0" exitCode=0
Oct 30 20:50:13.783677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:13.783299    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9" event={"ID":"3772bdaa-a3ac-441c-b46b-2dc0573937de","Type":"ContainerDied","Data":"fed4a782f13e9be23dcdd4f32a657e6c814e85ae194e1d852c417f88a91baae0"}
Oct 30 20:50:15.068649 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.068625    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:15.084897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.084842    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2\" (UniqueName: \"kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2\") pod \"3772bdaa-a3ac-441c-b46b-2dc0573937de\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") "
Oct 30 20:50:15.085016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.084914    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca\") pod \"3772bdaa-a3ac-441c-b46b-2dc0573937de\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") "
Oct 30 20:50:15.085085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.085073    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca\") pod \"3772bdaa-a3ac-441c-b46b-2dc0573937de\" (UID: \"3772bdaa-a3ac-441c-b46b-2dc0573937de\") "
Oct 30 20:50:15.085245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.085220    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca" pod "3772bdaa-a3ac-441c-b46b-2dc0573937de" (UID: "3772bdaa-a3ac-441c-b46b-2dc0573937de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:15.085552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.085538    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:15.086984 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.086963    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca" pod "3772bdaa-a3ac-441c-b46b-2dc0573937de" (UID: "3772bdaa-a3ac-441c-b46b-2dc0573937de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:15.087067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.087001    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2" pod "3772bdaa-a3ac-441c-b46b-2dc0573937de" (UID: "3772bdaa-a3ac-441c-b46b-2dc0573937de"). InnerVolumeSpecName "kube-api-access-wrjl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:15.186726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.186698    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3772bdaa-a3ac-441c-b46b-2dc0573937de-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:15.186726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.186727    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjl2\" (UniqueName: \"kubernetes.io/projected/3772bdaa-a3ac-441c-b46b-2dc0573937de-kube-api-access-wrjl2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:15.830168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.830092    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"
Oct 30 20:50:15.830168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.830127    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9" event={"ID":"3772bdaa-a3ac-441c-b46b-2dc0573937de","Type":"ContainerDied","Data":"a0b7cfeebb0962de861c47bcdac24b261f293b06cc47c11329a7a892c6422b99"}
Oct 30 20:50:15.830396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:15.830180    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b7cfeebb0962de861c47bcdac24b261f293b06cc47c11329a7a892c6422b99"
Oct 30 20:50:19.525072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.525024    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"]
Oct 30 20:50:19.578764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.574092    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy"
Oct 30 20:50:19.578983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.578962    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4947cbd-7f16-417b-9730-8b226f74a4f7" containerName="istio-proxy"
Oct 30 20:50:19.579188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.579175    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3772bdaa-a3ac-441c-b46b-2dc0573937de" containerName="revoke-keys"
Oct 30 20:50:19.579274 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.579263    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3772bdaa-a3ac-441c-b46b-2dc0573937de" containerName="revoke-keys"
Oct 30 20:50:19.583020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.582995    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3772bdaa-a3ac-441c-b46b-2dc0573937de" containerName="revoke-keys"
Oct 30 20:50:19.605681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.604665    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.741797 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.729550    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"]
Oct 30 20:50:19.742628 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.742605    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3f09ec-openshift-default-dockercfg-6mmdp\""
Oct 30 20:50:19.766459 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.766357    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb69b\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.766657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.766639    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768286    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768445    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768496    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768550    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768583    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768613    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.768764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.768636    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.869861 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.869756    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb69b\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.869883    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.869978    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870378    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870440    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870493    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870530    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870559    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.870795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870576    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.871003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.870979    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.871069 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.871030    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.871178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.871155    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.871244 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.871185    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.871378 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.871360    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.873223 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.873196    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.873532 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.873516    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.927256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.927226    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.927381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.927300    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb69b\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b\") pod \"e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") " pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:19.989822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:19.989797    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:20.450018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.449986    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"]
Oct 30 20:50:20.455313 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:50:20.455277    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f81c78_fe7a_4e77_afbb_fd21816db221.slice/crio-e4e31f67d5c6fd30ab4d0e968fd8d5e7c5a2e909fd67c393611a4167120c0b1d WatchSource:0}: Error finding container e4e31f67d5c6fd30ab4d0e968fd8d5e7c5a2e909fd67c393611a4167120c0b1d: Status 404 returned error can't find the container with id e4e31f67d5c6fd30ab4d0e968fd8d5e7c5a2e909fd67c393611a4167120c0b1d
Oct 30 20:50:20.458764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.458700    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:20.458899 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.458817    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:20.458899 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.458870    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:20.868130 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.868088    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" event={"ID":"45f81c78-fe7a-4e77-afbb-fd21816db221","Type":"ContainerStarted","Data":"de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"}
Oct 30 20:50:20.868641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.868156    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" event={"ID":"45f81c78-fe7a-4e77-afbb-fd21816db221","Type":"ContainerStarted","Data":"e4e31f67d5c6fd30ab4d0e968fd8d5e7c5a2e909fd67c393611a4167120c0b1d"}
Oct 30 20:50:20.987338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.987270    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podStartSLOduration=1.987250951 podStartE2EDuration="1.987250951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:20.985777638 +0000 UTC m=+1801.586579600" watchObservedRunningTime="2026-10-30 20:50:20.987250951 +0000 UTC m=+1801.588052911"
Oct 30 20:50:20.990262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:20.990234    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:21.306513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:21.306481    5967 scope.go:122] "RemoveContainer" containerID="6dd51ea01f6e4ff609c7a04d8b877f5851b3241a0ca2b407970d9c0510f37ab4"
Oct 30 20:50:21.317277 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:21.317261    5967 scope.go:122] "RemoveContainer" containerID="91d7bf4215bf2685a5a9cb268ac91833edd355849b8642516a31c1b0f6b3afca"
Oct 30 20:50:21.326323 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:21.326305    5967 scope.go:122] "RemoveContainer" containerID="279e622353f29c45b76084b4289d332ec3de3220188f5f1c39ab04f685ec04da"
Oct 30 20:50:21.334907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:21.334891    5967 scope.go:122] "RemoveContainer" containerID="f72db77776e2b95e045006e00a37f1269dce355f52c5772361bf7235131bb87e"
Oct 30 20:50:21.996406 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:21.996380    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:22.882150 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:22.882121    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:22.883133 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:22.883115    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:50:31.045816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.040932    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"]
Oct 30 20:50:31.065769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.065140    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.135297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.135261    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"]
Oct 30 20:50:31.142492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.142464    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-712099-openshift-default-dockercfg-pr8qg\""
Oct 30 20:50:31.178574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.177731    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.178993 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.178969    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.179207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.179187    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.179380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.179362    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.179566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.179550    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6sn\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.179795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.179777    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.180051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.180036    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.180199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.180185    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.180439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.180365    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282062    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282266    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282296    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282381    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282527    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282566    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282592    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282618    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.282909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282672    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6sn\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.283025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.282907    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.283671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.283561    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.284187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.283900    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.287213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.286337    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.287213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.286520    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.287425 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.287299    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.288995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.288972    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.346488 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.346421    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6sn\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.360288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.360262    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token\") pod \"e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") " pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.440496 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.440465    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:31.885723 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:50:31.885697    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a34841_d0e2_40dd_b4cc_8343e42b6da2.slice/crio-4fce32b064f93dca19589cb56b5c3109e68ebf6fee47bf4662a4c278d31704ff WatchSource:0}: Error finding container 4fce32b064f93dca19589cb56b5c3109e68ebf6fee47bf4662a4c278d31704ff: Status 404 returned error can't find the container with id 4fce32b064f93dca19589cb56b5c3109e68ebf6fee47bf4662a4c278d31704ff
Oct 30 20:50:31.886206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.886177    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"]
Oct 30 20:50:31.888563 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.888535    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:31.888628 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.888594    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:31.888628 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.888621    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:31.948991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:31.948949    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" event={"ID":"25a34841-d0e2-40dd-b4cc-8343e42b6da2","Type":"ContainerStarted","Data":"4fce32b064f93dca19589cb56b5c3109e68ebf6fee47bf4662a4c278d31704ff"}
Oct 30 20:50:31.986098 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:50:31.986039    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:50:32.958232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:32.958196    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" event={"ID":"25a34841-d0e2-40dd-b4cc-8343e42b6da2","Type":"ContainerStarted","Data":"7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"}
Oct 30 20:50:33.130852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:33.130800    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podStartSLOduration=3.130782058 podStartE2EDuration="3.130782058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:33.128128336 +0000 UTC m=+1813.728930290" watchObservedRunningTime="2026-10-30 20:50:33.130782058 +0000 UTC m=+1813.731584016"
Oct 30 20:50:33.441614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:33.441584    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:33.447201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:33.447178    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:33.965808 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:33.965781    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:50:33.966735 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:50:33.966718    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:51:01.980917 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:51:01.980877    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:51:13.516724 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.516682    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"]
Oct 30 20:51:13.521019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.521000    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.579541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.579514    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"]
Oct 30 20:51:13.624094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.624062    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:51:13.624241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.624122    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:51:13.624241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.624180    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:13.624241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.624188    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:51:13.640404 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.640380    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.640818 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.640795    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfldt\" (UniqueName: \"kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.640952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.640914    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.742525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.742495    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfldt\" (UniqueName: \"kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.742676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.742584    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.742676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.742638    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.743397 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.743331    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.745116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.745096    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.799859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.799803    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfldt\" (UniqueName: \"kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-68cgb\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:13.835923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:13.835896    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:14.213812 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:14.213786    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"]
Oct 30 20:51:14.215088 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:51:14.215066    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae3d458_9a41_4e87_9044_ac0cb1a99a24.slice/crio-1697ecf8ccc089c19519baa5c9479122d76cabdbcc561e05c88857280628591c WatchSource:0}: Error finding container 1697ecf8ccc089c19519baa5c9479122d76cabdbcc561e05c88857280628591c: Status 404 returned error can't find the container with id 1697ecf8ccc089c19519baa5c9479122d76cabdbcc561e05c88857280628591c
Oct 30 20:51:14.323172 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:14.323146    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb" event={"ID":"5ae3d458-9a41-4e87-9044-ac0cb1a99a24","Type":"ContainerStarted","Data":"1697ecf8ccc089c19519baa5c9479122d76cabdbcc561e05c88857280628591c"}
Oct 30 20:51:15.333422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:15.333345    5967 generic.go:356] "Generic (PLEG): container finished" podID="5ae3d458-9a41-4e87-9044-ac0cb1a99a24" containerID="e268aee4af61fb57c353256fc06db368374cb13105270c3a7ec8fe825006f71e" exitCode=0
Oct 30 20:51:15.333826 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:15.333428    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb" event={"ID":"5ae3d458-9a41-4e87-9044-ac0cb1a99a24","Type":"ContainerDied","Data":"e268aee4af61fb57c353256fc06db368374cb13105270c3a7ec8fe825006f71e"}
Oct 30 20:51:16.539205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.539182    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:16.578977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.578956    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca\") pod \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") "
Oct 30 20:51:16.579095 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.579031    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca\") pod \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") "
Oct 30 20:51:16.579190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.579178    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt\" (UniqueName: \"kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt\") pod \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\" (UID: \"5ae3d458-9a41-4e87-9044-ac0cb1a99a24\") "
Oct 30 20:51:16.579280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.579248    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca" pod "5ae3d458-9a41-4e87-9044-ac0cb1a99a24" (UID: "5ae3d458-9a41-4e87-9044-ac0cb1a99a24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:16.579603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.579588    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:16.581138 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.581116    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca" pod "5ae3d458-9a41-4e87-9044-ac0cb1a99a24" (UID: "5ae3d458-9a41-4e87-9044-ac0cb1a99a24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:16.581234 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.581223    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt" pod "5ae3d458-9a41-4e87-9044-ac0cb1a99a24" (UID: "5ae3d458-9a41-4e87-9044-ac0cb1a99a24"). InnerVolumeSpecName "kube-api-access-xfldt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:16.680898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.680869    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfldt\" (UniqueName: \"kubernetes.io/projected/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-kube-api-access-xfldt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:16.680898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:16.680897    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3d458-9a41-4e87-9044-ac0cb1a99a24-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:17.349809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:17.349779    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"
Oct 30 20:51:17.349809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:17.349797    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb" event={"ID":"5ae3d458-9a41-4e87-9044-ac0cb1a99a24","Type":"ContainerDied","Data":"1697ecf8ccc089c19519baa5c9479122d76cabdbcc561e05c88857280628591c"}
Oct 30 20:51:17.350071 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:17.349858    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1697ecf8ccc089c19519baa5c9479122d76cabdbcc561e05c88857280628591c"
Oct 30 20:51:21.402767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:21.402213    5967 scope.go:122] "RemoveContainer" containerID="8adc90ef2f9f2715b8ac180a2d7d166774299e484ebfa937793a1edf6c0163af"
Oct 30 20:51:29.472232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.471358    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:50510->10.134.0.161:15021: read: connection reset by peer" start-of-body=
Oct 30 20:51:29.472232 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.471435    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:50510->10.134.0.161:15021: read: connection reset by peer"
Oct 30 20:51:29.519692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.519665    5967 generic.go:356] "Generic (PLEG): container finished" podID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerID="6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075" exitCode=0
Oct 30 20:51:29.519829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.519730    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" event={"ID":"b6926a1c-820c-48b6-b3fd-31b4026a7804","Type":"ContainerDied","Data":"6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"}
Oct 30 20:51:29.520177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.520162    5967 scope.go:122] "RemoveContainer" containerID="6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"
Oct 30 20:51:29.521553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.521536    5967 generic.go:356] "Generic (PLEG): container finished" podID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerID="a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575" exitCode=0
Oct 30 20:51:29.521637 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.521611    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" event={"ID":"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031","Type":"ContainerDied","Data":"a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"}
Oct 30 20:51:29.521807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.521779    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:29.521860 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.521849    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:29.521895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.521884    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:29.521978 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.521964    5967 scope.go:122] "RemoveContainer" containerID="a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"
Oct 30 20:51:29.523394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.523367    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:29.523464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.523429    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:29.523464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:29.523457    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:30.539160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:30.539124    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" event={"ID":"b6926a1c-820c-48b6-b3fd-31b4026a7804","Type":"ContainerStarted","Data":"3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0"}
Oct 30 20:51:30.541350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:30.541324    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" event={"ID":"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031","Type":"ContainerStarted","Data":"4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc"}
Oct 30 20:51:30.909502 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:30.909354    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:51:30.919905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:30.919866    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:51:30.920182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:30.920151    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 20:51:31.014764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:31.014191    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:51:31.014764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:31.014339    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 20:51:31.018768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:31.014970    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 20:51:31.897767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:31.897046    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:51:31.897767 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:31.897114    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 20:51:32.015766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:32.013440    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 20:51:32.015766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:32.013511    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 20:51:32.028133 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:51:32.028099    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:51:32.896764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:32.895833    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:51:32.896764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:32.895893    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 20:51:33.008763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:33.008267    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 20:51:33.008763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:33.008324    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 20:51:33.912764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:33.912084    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:51:33.912764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:33.912144    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 20:51:34.012766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:34.012063    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 20:51:34.012766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:34.012121    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 20:51:34.224169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:34.223879    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:51:34.896121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:34.896084    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:51:34.896394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:34.896371    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 20:51:35.008119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:35.008083    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 20:51:35.008420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:35.008393    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 20:51:35.896766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:35.896335    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:51:35.896766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:35.896394    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 20:51:36.006572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:36.006528    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 20:51:36.006833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:36.006587    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 20:51:36.899978 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:36.899943    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:51:36.901297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:36.901272    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:51:37.010051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:37.010023    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:51:37.010295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:37.010277    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:51:37.010907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:37.010890    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:51:53.300167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.300138    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"]
Oct 30 20:51:53.300848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.300833    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5ae3d458-9a41-4e87-9044-ac0cb1a99a24" containerName="revoke-keys"
Oct 30 20:51:53.300903 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.300849    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae3d458-9a41-4e87-9044-ac0cb1a99a24" containerName="revoke-keys"
Oct 30 20:51:53.300983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.300974    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae3d458-9a41-4e87-9044-ac0cb1a99a24" containerName="revoke-keys"
Oct 30 20:51:53.303852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.303835    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.356602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.356579    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3f0723-openshift-default-dockercfg-pmw7m\""
Oct 30 20:51:53.383018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.382984    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-684bq\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.383181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.383123    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.383271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.383252    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.383349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.383292    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.383573 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.383549    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.383787 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.383761    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.383945 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.383926    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.384015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.384003    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.384074 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.384062    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.410035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.410014    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"]
Oct 30 20:51:53.519908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.519872    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.520301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520275    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-684bq\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.520511 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520281    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520513    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520708    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520764    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520807    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520815    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.520864    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.521015    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.521019    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.521109    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.521602    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.523348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.521999    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.553176 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.553095    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.570871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.570844    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.589766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.588220    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.596841 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.595908    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-684bq\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq\") pod \"e2e-api-a-3f0723-openshift-default-898ff695-vl5s7\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") " pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:53.616960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:53.616922    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:54.099051 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.099018    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"]
Oct 30 20:51:54.131045 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:51:54.131014    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a7b15_288a_4b84_ac8f_b179c382dc9e.slice/crio-e95bae723129b075f7dcc4e9619251443a69fafa3a935e1e45505dbb17ba72d4 WatchSource:0}: Error finding container e95bae723129b075f7dcc4e9619251443a69fafa3a935e1e45505dbb17ba72d4: Status 404 returned error can't find the container with id e95bae723129b075f7dcc4e9619251443a69fafa3a935e1e45505dbb17ba72d4
Oct 30 20:51:54.133756 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.133713    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:54.133832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.133821    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:54.133879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.133863    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:54.792100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.792057    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" event={"ID":"ce9a7b15-288a-4b84-ac8f-b179c382dc9e","Type":"ContainerStarted","Data":"8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"}
Oct 30 20:51:54.792510 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.792123    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" event={"ID":"ce9a7b15-288a-4b84-ac8f-b179c382dc9e","Type":"ContainerStarted","Data":"e95bae723129b075f7dcc4e9619251443a69fafa3a935e1e45505dbb17ba72d4"}
Oct 30 20:51:54.923032 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:54.922982    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podStartSLOduration=1.922967463 podStartE2EDuration="1.922967463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:54.918340761 +0000 UTC m=+1895.519142720" watchObservedRunningTime="2026-10-30 20:51:54.922967463 +0000 UTC m=+1895.523769423"
Oct 30 20:51:55.617802 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:55.617770    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:55.623416 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:55.623389    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:55.801549 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:55.801520    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:51:55.802536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:51:55.802517    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:52:01.981789 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:52:01.981702    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:52:06.913892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:06.913860    5967 generic.go:356] "Generic (PLEG): container finished" podID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerID="b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6" exitCode=0
Oct 30 20:52:06.914315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:06.913932    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" event={"ID":"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2","Type":"ContainerDied","Data":"b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"}
Oct 30 20:52:06.914360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:06.914324    5967 scope.go:122] "RemoveContainer" containerID="b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"
Oct 30 20:52:06.916016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:06.915988    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:06.916119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:06.916057    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:06.916119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:06.916089    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:07.931723 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:07.931687    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" event={"ID":"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2","Type":"ContainerStarted","Data":"97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa"}
Oct 30 20:52:08.517475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:08.517446    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:52:08.517712 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:08.517694    5967 patch_prober.go:29] interesting pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:52:08.517817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:08.517784    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 20:52:09.518233 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:09.518191    5967 patch_prober.go:29] interesting pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:52:09.518632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:09.518265    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 20:52:10.518228 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:10.518196    5967 patch_prober.go:29] interesting pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:52:10.518442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:10.518258    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 20:52:11.517411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:11.517378    5967 patch_prober.go:29] interesting pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:52:11.517592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:11.517435    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 20:52:12.517415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:12.517383    5967 patch_prober.go:29] interesting pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 20:52:12.517837 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:12.517433    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 20:52:13.521929 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:13.521900    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:52:13.522428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:13.522196    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:52:13.522955 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:13.522935    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:52:21.449646 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:21.449615    5967 scope.go:122] "RemoveContainer" containerID="0113fb3a13bced40ec2de394a87ff6599a6c9b3b7272dd6c6843abaf5be8d16a"
Oct 30 20:52:21.459907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:21.459861    5967 scope.go:122] "RemoveContainer" containerID="fa0761b6b56d2fe5af636e83fa239cb817f109ea433f6e6e4e4e36539d0c3959"
Oct 30 20:52:24.952047 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:24.952004    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"]
Oct 30 20:52:25.065681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.065648    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"]
Oct 30 20:52:25.065991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.065806    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.118240 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.118214    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:25.119289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.118577    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:52:25.119473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.118618    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:52:25.119600 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.119171    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:52:25.154535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.154501    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.154704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.154577    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.154704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.154620    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2s9\" (UniqueName: \"kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.256108 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.256082    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.256317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.256133    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2s9\" (UniqueName: \"kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.256317 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.256270    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.256853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.256832    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.258493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.258476    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.312543 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.312511    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2s9\" (UniqueName: \"kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9\") pod \"maas-api-revoke-sub-2486e8df-m95xb\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.389998 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.389958    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:25.786751 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:25.786685    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"]
Oct 30 20:52:25.787822 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:52:25.787793    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ac5c9f_5f94_4f12_81c9_97b0964c532d.slice/crio-0bf46fb3202a82a7a715f25132865d050a5c489ff4fd89fe6f02e0faab03969d WatchSource:0}: Error finding container 0bf46fb3202a82a7a715f25132865d050a5c489ff4fd89fe6f02e0faab03969d: Status 404 returned error can't find the container with id 0bf46fb3202a82a7a715f25132865d050a5c489ff4fd89fe6f02e0faab03969d
Oct 30 20:52:26.149415 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:26.149342    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb" event={"ID":"92ac5c9f-5f94-4f12-81c9-97b0964c532d","Type":"ContainerStarted","Data":"0bf46fb3202a82a7a715f25132865d050a5c489ff4fd89fe6f02e0faab03969d"}
Oct 30 20:52:27.160868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:27.160776    5967 generic.go:356] "Generic (PLEG): container finished" podID="92ac5c9f-5f94-4f12-81c9-97b0964c532d" containerID="7612ff1377d000c213826a036e6d6a17e6b3d0cdbc57abf1bf58b36d9444a17b" exitCode=0
Oct 30 20:52:27.161297 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:27.160873    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb" event={"ID":"92ac5c9f-5f94-4f12-81c9-97b0964c532d","Type":"ContainerDied","Data":"7612ff1377d000c213826a036e6d6a17e6b3d0cdbc57abf1bf58b36d9444a17b"}
Oct 30 20:52:28.340944 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.340920    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:28.510533 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.510484    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca\") pod \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") "
Oct 30 20:52:28.510713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.510632    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9\" (UniqueName: \"kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9\") pod \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") "
Oct 30 20:52:28.510713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.510678    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca\") pod \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\" (UID: \"92ac5c9f-5f94-4f12-81c9-97b0964c532d\") "
Oct 30 20:52:28.510980 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.510951    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca" pod "92ac5c9f-5f94-4f12-81c9-97b0964c532d" (UID: "92ac5c9f-5f94-4f12-81c9-97b0964c532d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:28.511466 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.511440    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:28.513278 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.513254    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca" pod "92ac5c9f-5f94-4f12-81c9-97b0964c532d" (UID: "92ac5c9f-5f94-4f12-81c9-97b0964c532d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:28.513373 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.513300    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9" pod "92ac5c9f-5f94-4f12-81c9-97b0964c532d" (UID: "92ac5c9f-5f94-4f12-81c9-97b0964c532d"). InnerVolumeSpecName "kube-api-access-9q2s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:28.612953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.612924    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q2s9\" (UniqueName: \"kubernetes.io/projected/92ac5c9f-5f94-4f12-81c9-97b0964c532d-kube-api-access-9q2s9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:28.612953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:28.612950    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac5c9f-5f94-4f12-81c9-97b0964c532d-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:29.176078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:29.176047    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"
Oct 30 20:52:29.176272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:29.176082    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb" event={"ID":"92ac5c9f-5f94-4f12-81c9-97b0964c532d","Type":"ContainerDied","Data":"0bf46fb3202a82a7a715f25132865d050a5c489ff4fd89fe6f02e0faab03969d"}
Oct 30 20:52:29.176272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:29.176131    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf46fb3202a82a7a715f25132865d050a5c489ff4fd89fe6f02e0faab03969d"
Oct 30 20:52:30.186349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.186302    5967 generic.go:356] "Generic (PLEG): container finished" podID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerID="de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f" exitCode=0
Oct 30 20:52:30.186879 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.186382    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" event={"ID":"45f81c78-fe7a-4e77-afbb-fd21816db221","Type":"ContainerDied","Data":"de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"}
Oct 30 20:52:30.186937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.186913    5967 scope.go:122] "RemoveContainer" containerID="de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"
Oct 30 20:52:30.188805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.188768    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:30.188957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.188852    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:30.188957 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.188905    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:30.413680 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.413649    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"]
Oct 30 20:52:30.413921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.413895    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy" containerID="cri-o://a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9" gracePeriod=30
Oct 30 20:52:30.475235 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.475201    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"]
Oct 30 20:52:30.475460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.475429    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy" containerID="cri-o://12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089" gracePeriod=30
Oct 30 20:52:30.479253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.479224    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"]
Oct 30 20:52:30.479477 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:30.479447    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" containerID="cri-o://36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a" gracePeriod=30
Oct 30 20:52:31.202653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.202619    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" event={"ID":"45f81c78-fe7a-4e77-afbb-fd21816db221","Type":"ContainerStarted","Data":"fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37"}
Oct 30 20:52:31.330192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.330157    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"]
Oct 30 20:52:31.330985 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.330969    5967 cpu_manager.go:414] "containerMap: removing container" podUID="92ac5c9f-5f94-4f12-81c9-97b0964c532d" containerName="revoke-keys"
Oct 30 20:52:31.331033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.330987    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ac5c9f-5f94-4f12-81c9-97b0964c532d" containerName="revoke-keys"
Oct 30 20:52:31.331121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.331112    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ac5c9f-5f94-4f12-81c9-97b0964c532d" containerName="revoke-keys"
Oct 30 20:52:31.333669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.333650    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.392989 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.392058    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:31.396141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.395584    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:52:31.396141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.395700    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:52:31.396141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.395895    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:52:31.400226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.400166    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"]
Oct 30 20:52:31.432201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.432154    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.432381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.432362    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.432534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.432510    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt59g\" (UniqueName: \"kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.534138 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.534086    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.534350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.534205    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt59g\" (UniqueName: \"kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.534433 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.534385    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.534779 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.534723    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.537015 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.536988    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.596988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.596947    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt59g\" (UniqueName: \"kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.646631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.646595    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:31.995762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.995707    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:52:31.995961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.995777    5967 patch_prober.go:29] interesting pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:52:31.995961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:31.995831    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 20:52:32.004257 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:52:32.004226    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:52:32.060342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:32.060317    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"]
Oct 30 20:52:32.061342 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:52:32.061317    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d98592_cf77_413f_8a64_f044af899f58.slice/crio-2d312fa0d1b8111e97abbf1bf018125d484978b3df72e2e294209da04c08278b WatchSource:0}: Error finding container 2d312fa0d1b8111e97abbf1bf018125d484978b3df72e2e294209da04c08278b: Status 404 returned error can't find the container with id 2d312fa0d1b8111e97abbf1bf018125d484978b3df72e2e294209da04c08278b
Oct 30 20:52:32.211143 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:32.211106    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w" event={"ID":"84d98592-cf77-413f-8a64-f044af899f58","Type":"ContainerStarted","Data":"2d312fa0d1b8111e97abbf1bf018125d484978b3df72e2e294209da04c08278b"}
Oct 30 20:52:32.994978 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:32.994920    5967 patch_prober.go:29] interesting pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:52:32.995286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:32.994973    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 20:52:33.220112 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:33.220077    5967 generic.go:356] "Generic (PLEG): container finished" podID="84d98592-cf77-413f-8a64-f044af899f58" containerID="f2b844f3300c0178304f4f7127724bede2f66794118d419617e3e15492e77dea" exitCode=0
Oct 30 20:52:33.220506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:33.220164    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w" event={"ID":"84d98592-cf77-413f-8a64-f044af899f58","Type":"ContainerDied","Data":"f2b844f3300c0178304f4f7127724bede2f66794118d419617e3e15492e77dea"}
Oct 30 20:52:33.715367 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:33.715336    5967 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:52:33.715541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:33.715387    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:52:34.001822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.001307    5967 patch_prober.go:29] interesting pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:52:34.001822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.001377    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 20:52:34.430122 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.430097    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:34.611273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.611190    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca\") pod \"84d98592-cf77-413f-8a64-f044af899f58\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") "
Oct 30 20:52:34.611442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.611273    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g\" (UniqueName: \"kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g\") pod \"84d98592-cf77-413f-8a64-f044af899f58\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") "
Oct 30 20:52:34.611442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.611425    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca\") pod \"84d98592-cf77-413f-8a64-f044af899f58\" (UID: \"84d98592-cf77-413f-8a64-f044af899f58\") "
Oct 30 20:52:34.611624 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.611498    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca" pod "84d98592-cf77-413f-8a64-f044af899f58" (UID: "84d98592-cf77-413f-8a64-f044af899f58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:34.612301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.612274    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d98592-cf77-413f-8a64-f044af899f58-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:34.613617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.613587    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca" pod "84d98592-cf77-413f-8a64-f044af899f58" (UID: "84d98592-cf77-413f-8a64-f044af899f58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:34.613727 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.613703    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g" pod "84d98592-cf77-413f-8a64-f044af899f58" (UID: "84d98592-cf77-413f-8a64-f044af899f58"). InnerVolumeSpecName "kube-api-access-vt59g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:34.713772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.713709    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt59g\" (UniqueName: \"kubernetes.io/projected/84d98592-cf77-413f-8a64-f044af899f58-kube-api-access-vt59g\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:34.713772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.713766    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d98592-cf77-413f-8a64-f044af899f58-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:34.994084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.994045    5967 patch_prober.go:29] interesting pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:52:34.994287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:34.994111    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 20:52:35.235592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.235563    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"
Oct 30 20:52:35.235771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.235595    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w" event={"ID":"84d98592-cf77-413f-8a64-f044af899f58","Type":"ContainerDied","Data":"2d312fa0d1b8111e97abbf1bf018125d484978b3df72e2e294209da04c08278b"}
Oct 30 20:52:35.235771 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.235642    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d312fa0d1b8111e97abbf1bf018125d484978b3df72e2e294209da04c08278b"
Oct 30 20:52:35.432170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.432143    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"]
Oct 30 20:52:35.432862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.432847    5967 cpu_manager.go:414] "containerMap: removing container" podUID="84d98592-cf77-413f-8a64-f044af899f58" containerName="revoke-keys"
Oct 30 20:52:35.432920 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.432863    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d98592-cf77-413f-8a64-f044af899f58" containerName="revoke-keys"
Oct 30 20:52:35.433005 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.432996    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d98592-cf77-413f-8a64-f044af899f58" containerName="revoke-keys"
Oct 30 20:52:35.435775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.435756    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.490775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.488766    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:52:35.490775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.488948    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:52:35.490775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.489057    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:52:35.490775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.489609    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:35.547750 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.547719    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"]
Oct 30 20:52:35.627060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.626953    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.627060 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.627028    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl476\" (UniqueName: \"kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.627365 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.627304    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.728897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.728869    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.729072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.728933    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl476\" (UniqueName: \"kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.729194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.729176    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.729921 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.729893    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.731850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.731672    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.806545 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.806483    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl476\" (UniqueName: \"kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.841288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.841022    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:52:35.863537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.863512    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:52:35.927673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.927650    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:52:35.931245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.931218    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:35.944410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944379    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.944546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944436    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.944546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944469    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.944546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944519    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.944696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944545    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.944696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944585    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.944696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944609    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.944696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944644    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.944696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944670    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.944969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944708    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.944969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944772    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.944969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944826    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.945152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.945131    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.946023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.945975    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.946371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.946122    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.946371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.946278    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.946730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.946710    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.946859 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.946770    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.947146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.947125    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.947645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.947611    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.950269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.950246    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "kube-api-access-wz29c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.952922 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.952893    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.954636 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.954608    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "kube-api-access-zr9jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.955571 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.944859    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.955663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955629    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.955726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955693    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.955833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955819    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.955887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955870    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.955939 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955903    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.955997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955950    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.955997 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.955983    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.956102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956038    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.956102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956070    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.956206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956110    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.956206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956142    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.956206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956191    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v\") pod \"ebaf2514-421d-4d33-9e82-1559bf65c464\" (UID: \"ebaf2514-421d-4d33-9e82-1559bf65c464\") "
Oct 30 20:52:35.956473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956254    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data\") pod \"42bf4655-d839-4ac7-a0bc-6cedc767c039\" (UID: \"42bf4655-d839-4ac7-a0bc-6cedc767c039\") "
Oct 30 20:52:35.956473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956301    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy\") pod \"2c3c959b-7b4f-4cbd-84af-904623075aa0\" (UID: \"2c3c959b-7b4f-4cbd-84af-904623075aa0\") "
Oct 30 20:52:35.956952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956934    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaf2514-421d-4d33-9e82-1559bf65c464-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956958    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956972    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c959b-7b4f-4cbd-84af-904623075aa0-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956985    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.956998    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz29c\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-kube-api-access-wz29c\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.957012    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr9jw\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-kube-api-access-zr9jw\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.957025    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.957038    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.957051    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.957065    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf4655-d839-4ac7-a0bc-6cedc767c039-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.957348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.957080    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.958423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.958396    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.958605 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.958582    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.958882 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.958858    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.959197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.959117    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.959197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.959173    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.959582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.959342    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.960760 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.960710    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.960935 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.960917    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.961948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.961924    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.962775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.962721    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.962907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.962883    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.963103 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.963072    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.963870 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.963830    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.965252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.965118    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy" pod "42bf4655-d839-4ac7-a0bc-6cedc767c039" (UID: "42bf4655-d839-4ac7-a0bc-6cedc767c039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.965252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.965185    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo" pod "2c3c959b-7b4f-4cbd-84af-904623075aa0" (UID: "2c3c959b-7b4f-4cbd-84af-904623075aa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.965252 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:35.965237    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v" pod "ebaf2514-421d-4d33-9e82-1559bf65c464" (UID: "ebaf2514-421d-4d33-9e82-1559bf65c464"). InnerVolumeSpecName "kube-api-access-f7x7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:36.006820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.006687    5967 patch_prober.go:29] interesting pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:52:36.006820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.006780    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061173    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061203    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061218    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061231    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061243    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061256    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061270    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061300    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061314    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061327    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061341    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7x7v\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-kube-api-access-f7x7v\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061355    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061367    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3c959b-7b4f-4cbd-84af-904623075aa0-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061381    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf4655-d839-4ac7-a0bc-6cedc767c039-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061394    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.061590 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.061407    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaf2514-421d-4d33-9e82-1559bf65c464-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.249668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.249542    5967 generic.go:356] "Generic (PLEG): container finished" podID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerID="36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a" exitCode=0
Oct 30 20:52:36.249668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.249602    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"
Oct 30 20:52:36.249668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.249622    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" event={"ID":"ebaf2514-421d-4d33-9e82-1559bf65c464","Type":"ContainerDied","Data":"36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a"}
Oct 30 20:52:36.250146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.249727    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj" event={"ID":"ebaf2514-421d-4d33-9e82-1559bf65c464","Type":"ContainerDied","Data":"63932b02f9806d2f21c5b772a38a6b9274d2868f57a7a250a54c704eba56a83d"}
Oct 30 20:52:36.250146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.249774    5967 scope.go:122] "RemoveContainer" containerID="36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a"
Oct 30 20:52:36.266964 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.266929    5967 generic.go:356] "Generic (PLEG): container finished" podID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerID="12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089" exitCode=0
Oct 30 20:52:36.267183 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.267155    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" event={"ID":"2c3c959b-7b4f-4cbd-84af-904623075aa0","Type":"ContainerDied","Data":"12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089"}
Oct 30 20:52:36.267270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.267229    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7" event={"ID":"2c3c959b-7b4f-4cbd-84af-904623075aa0","Type":"ContainerDied","Data":"5bb183e1df76a25046e38b1720d643644264a988fa385097b841df6d56648eba"}
Oct 30 20:52:36.267337 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.267274    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"
Oct 30 20:52:36.270114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.269843    5967 generic.go:356] "Generic (PLEG): container finished" podID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerID="a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9" exitCode=0
Oct 30 20:52:36.270114 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.269943    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" event={"ID":"42bf4655-d839-4ac7-a0bc-6cedc767c039","Type":"ContainerDied","Data":"a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9"}
Oct 30 20:52:36.270842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.270309    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt" event={"ID":"42bf4655-d839-4ac7-a0bc-6cedc767c039","Type":"ContainerDied","Data":"e1f2635ed5213c01a7084ae34cc9dcab1da59df96c15f0b1e088db44e4463f7c"}
Oct 30 20:52:36.270842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.270403    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"
Oct 30 20:52:36.281668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.281581    5967 scope.go:122] "RemoveContainer" containerID="36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a"
Oct 30 20:52:36.281962 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:52:36.281935    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a\": container with ID starting with 36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a not found: ID does not exist" containerID="36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a"
Oct 30 20:52:36.282041 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.281973    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a"} err="failed to get container status \"36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a\": rpc error: code = NotFound desc = could not find container \"36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a\": container with ID starting with 36746bc350a631edde1d4f81badc0659c7e3a394a35bea59a08c44d3115d8b4a not found: ID does not exist"
Oct 30 20:52:36.282041 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.281996    5967 scope.go:122] "RemoveContainer" containerID="12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089"
Oct 30 20:52:36.295529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.295503    5967 scope.go:122] "RemoveContainer" containerID="12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089"
Oct 30 20:52:36.295829 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:52:36.295810    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089\": container with ID starting with 12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089 not found: ID does not exist" containerID="12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089"
Oct 30 20:52:36.295926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.295839    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089"} err="failed to get container status \"12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089\": rpc error: code = NotFound desc = could not find container \"12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089\": container with ID starting with 12c2d40801f7e66aa34266f2b1e4f91c33562176fda412774d6720024abe9089 not found: ID does not exist"
Oct 30 20:52:36.295926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.295855    5967 scope.go:122] "RemoveContainer" containerID="a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9"
Oct 30 20:52:36.305394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.305366    5967 scope.go:122] "RemoveContainer" containerID="a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9"
Oct 30 20:52:36.305655 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:52:36.305629    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9\": container with ID starting with a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9 not found: ID does not exist" containerID="a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9"
Oct 30 20:52:36.305754 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.305654    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9"} err="failed to get container status \"a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9\": rpc error: code = NotFound desc = could not find container \"a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9\": container with ID starting with a4464736f51cd38907fca47554051fd20363ff615dfc57581cb8f1835998f4d9 not found: ID does not exist"
Oct 30 20:52:36.380518 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.380490    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"]
Oct 30 20:52:36.382000 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:52:36.381973    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd987ceb1_fbe3_435c_99a1_211124844bbc.slice/crio-1dd7eb40c239fa3e35c719087bd50cdc2d69e6d45df1621a6c85be88f1041c1c WatchSource:0}: Error finding container 1dd7eb40c239fa3e35c719087bd50cdc2d69e6d45df1621a6c85be88f1041c1c: Status 404 returned error can't find the container with id 1dd7eb40c239fa3e35c719087bd50cdc2d69e6d45df1621a6c85be88f1041c1c
Oct 30 20:52:36.417853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.417831    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"]
Oct 30 20:52:36.421094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.421073    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt"]
Oct 30 20:52:36.528610 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.528533    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"]
Oct 30 20:52:36.531949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.531925    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj"]
Oct 30 20:52:36.640559 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.640534    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"]
Oct 30 20:52:36.643624 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.643604    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7"]
Oct 30 20:52:36.995250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.995224    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:52:36.995733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.995717    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:52:36.996400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:36.996382    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:52:37.287805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:37.287772    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2" event={"ID":"d987ceb1-fbe3-435c-99a1-211124844bbc","Type":"ContainerStarted","Data":"1a54b83d78aa9e1387baa8280e70a716538459fb995ca099284071b8bd25d7b5"}
Oct 30 20:52:37.287946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:37.287831    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2" event={"ID":"d987ceb1-fbe3-435c-99a1-211124844bbc","Type":"ContainerStarted","Data":"1dd7eb40c239fa3e35c719087bd50cdc2d69e6d45df1621a6c85be88f1041c1c"}
Oct 30 20:52:37.544013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:37.543940    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" path="/var/lib/kubelet/pods/2c3c959b-7b4f-4cbd-84af-904623075aa0/volumes"
Oct 30 20:52:37.544400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:37.544386    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" path="/var/lib/kubelet/pods/42bf4655-d839-4ac7-a0bc-6cedc767c039/volumes"
Oct 30 20:52:37.544836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:37.544822    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" path="/var/lib/kubelet/pods/ebaf2514-421d-4d33-9e82-1559bf65c464/volumes"
Oct 30 20:52:38.298338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.298307    5967 generic.go:356] "Generic (PLEG): container finished" podID="d987ceb1-fbe3-435c-99a1-211124844bbc" containerID="1a54b83d78aa9e1387baa8280e70a716538459fb995ca099284071b8bd25d7b5" exitCode=0
Oct 30 20:52:38.298516 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.298358    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2" event={"ID":"d987ceb1-fbe3-435c-99a1-211124844bbc","Type":"ContainerDied","Data":"1a54b83d78aa9e1387baa8280e70a716538459fb995ca099284071b8bd25d7b5"}
Oct 30 20:52:38.449632 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.449608    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:38.493781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.493729    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476\" (UniqueName: \"kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476\") pod \"d987ceb1-fbe3-435c-99a1-211124844bbc\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") "
Oct 30 20:52:38.493930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.493915    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca\") pod \"d987ceb1-fbe3-435c-99a1-211124844bbc\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") "
Oct 30 20:52:38.493991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.493977    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca\") pod \"d987ceb1-fbe3-435c-99a1-211124844bbc\" (UID: \"d987ceb1-fbe3-435c-99a1-211124844bbc\") "
Oct 30 20:52:38.494270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.494250    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca" pod "d987ceb1-fbe3-435c-99a1-211124844bbc" (UID: "d987ceb1-fbe3-435c-99a1-211124844bbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:38.494560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.494542    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:38.495900 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.495872    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476" pod "d987ceb1-fbe3-435c-99a1-211124844bbc" (UID: "d987ceb1-fbe3-435c-99a1-211124844bbc"). InnerVolumeSpecName "kube-api-access-zl476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:38.496072 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.496053    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca" pod "d987ceb1-fbe3-435c-99a1-211124844bbc" (UID: "d987ceb1-fbe3-435c-99a1-211124844bbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:38.597366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.597294    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl476\" (UniqueName: \"kubernetes.io/projected/d987ceb1-fbe3-435c-99a1-211124844bbc-kube-api-access-zl476\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:38.597889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:38.597858    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d987ceb1-fbe3-435c-99a1-211124844bbc-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:39.308042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:39.308014    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"
Oct 30 20:52:39.308042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:39.308022    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2" event={"ID":"d987ceb1-fbe3-435c-99a1-211124844bbc","Type":"ContainerDied","Data":"1dd7eb40c239fa3e35c719087bd50cdc2d69e6d45df1621a6c85be88f1041c1c"}
Oct 30 20:52:39.308308 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:39.308109    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dd7eb40c239fa3e35c719087bd50cdc2d69e6d45df1621a6c85be88f1041c1c"
Oct 30 20:52:54.442522 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:54.442418    5967 generic.go:356] "Generic (PLEG): container finished" podID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerID="7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725" exitCode=0
Oct 30 20:52:54.442522 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:54.442499    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" event={"ID":"25a34841-d0e2-40dd-b4cc-8343e42b6da2","Type":"ContainerDied","Data":"7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"}
Oct 30 20:52:54.443116 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:54.442893    5967 scope.go:122] "RemoveContainer" containerID="7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"
Oct 30 20:52:54.444501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:54.444473    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:54.444585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:54.444541    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:54.444585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:54.444578    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:55.451882 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:55.451851    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" event={"ID":"25a34841-d0e2-40dd-b4cc-8343e42b6da2","Type":"ContainerStarted","Data":"6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4"}
Oct 30 20:52:56.441479 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:56.441448    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:52:56.441666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:56.441575    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:52:56.441666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:56.441651    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:52:57.441333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:57.441300    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:52:57.441721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:57.441356    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:52:58.441135 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:58.441106    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:52:58.441321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:58.441155    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:52:59.441248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:59.441219    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:52:59.441634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:52:59.441266    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:53:00.441211 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:00.441172    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:53:00.441484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:00.441230    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:53:01.445653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:01.445624    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:53:01.446113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:01.446088    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:53:01.446842 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:01.446823    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:53:01.981336 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:53:01.981303    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:53:02.655093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655062    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"]
Oct 30 20:53:02.655722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655709    5967 cpu_manager.go:414] "containerMap: removing container" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655724    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655771    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d987ceb1-fbe3-435c-99a1-211124844bbc" containerName="revoke-keys"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655777    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d987ceb1-fbe3-435c-99a1-211124844bbc" containerName="revoke-keys"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655789    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655795    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655804    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy"
Oct 30 20:53:02.655811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655808    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy"
Oct 30 20:53:02.656018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655910    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebaf2514-421d-4d33-9e82-1559bf65c464" containerName="istio-proxy"
Oct 30 20:53:02.656018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655920    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3c959b-7b4f-4cbd-84af-904623075aa0" containerName="istio-proxy"
Oct 30 20:53:02.656018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655933    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="42bf4655-d839-4ac7-a0bc-6cedc767c039" containerName="istio-proxy"
Oct 30 20:53:02.656018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.655942    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d987ceb1-fbe3-435c-99a1-211124844bbc" containerName="revoke-keys"
Oct 30 20:53:02.663099 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.663079    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.716526 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.716505    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"]
Oct 30 20:53:02.761844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.761819    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:02.761844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.761817    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:02.762016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.761844    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:02.763971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.763958    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:53:02.843805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.843782    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz77b\" (UniqueName: \"kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.843946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.843922    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.843991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.843971    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.945905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.945832    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz77b\" (UniqueName: \"kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.946043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.945926    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.946043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.945968    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.946657 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.946636    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:02.948525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:02.948506    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:03.002758 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:03.002718    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz77b\" (UniqueName: \"kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b\") pod \"maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:03.277063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:03.277037    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:03.656191 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:03.656160    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"]
Oct 30 20:53:03.657102 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:53:03.657066    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf4660e_4ff1_45b7_8ddb_df867099ccec.slice/crio-3ff8344593dde3c5d1d0b1249fff8b2f86b8179bcbfbd4561ee97684dae27b42 WatchSource:0}: Error finding container 3ff8344593dde3c5d1d0b1249fff8b2f86b8179bcbfbd4561ee97684dae27b42: Status 404 returned error can't find the container with id 3ff8344593dde3c5d1d0b1249fff8b2f86b8179bcbfbd4561ee97684dae27b42
Oct 30 20:53:04.607302 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:04.607255    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc" event={"ID":"dbf4660e-4ff1-45b7-8ddb-df867099ccec","Type":"ContainerStarted","Data":"dea47d12708d6b34305294ed0ccc8d9ca23abd6e54a217d40374e51f01d1f51d"}
Oct 30 20:53:04.607589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:04.607550    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc" event={"ID":"dbf4660e-4ff1-45b7-8ddb-df867099ccec","Type":"ContainerStarted","Data":"3ff8344593dde3c5d1d0b1249fff8b2f86b8179bcbfbd4561ee97684dae27b42"}
Oct 30 20:53:04.723968 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:04.723918    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc" podStartSLOduration=2.723903443 podStartE2EDuration="2.723903443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:04.721775231 +0000 UTC m=+1965.322577192" watchObservedRunningTime="2026-10-30 20:53:04.723903443 +0000 UTC m=+1965.324705402"
Oct 30 20:53:05.615410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:05.615378    5967 generic.go:356] "Generic (PLEG): container finished" podID="dbf4660e-4ff1-45b7-8ddb-df867099ccec" containerID="dea47d12708d6b34305294ed0ccc8d9ca23abd6e54a217d40374e51f01d1f51d" exitCode=0
Oct 30 20:53:05.615603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:05.615469    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc" event={"ID":"dbf4660e-4ff1-45b7-8ddb-df867099ccec","Type":"ContainerDied","Data":"dea47d12708d6b34305294ed0ccc8d9ca23abd6e54a217d40374e51f01d1f51d"}
Oct 30 20:53:06.794830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.794807    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:06.804928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.804908    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca\") pod \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") "
Oct 30 20:53:06.805031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.805018    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b\" (UniqueName: \"kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b\") pod \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") "
Oct 30 20:53:06.805132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.805119    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca\") pod \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\" (UID: \"dbf4660e-4ff1-45b7-8ddb-df867099ccec\") "
Oct 30 20:53:06.805208 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.805186    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca" pod "dbf4660e-4ff1-45b7-8ddb-df867099ccec" (UID: "dbf4660e-4ff1-45b7-8ddb-df867099ccec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:06.805563 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.805545    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:06.807100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.807080    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca" pod "dbf4660e-4ff1-45b7-8ddb-df867099ccec" (UID: "dbf4660e-4ff1-45b7-8ddb-df867099ccec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:06.807183 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.807140    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b" pod "dbf4660e-4ff1-45b7-8ddb-df867099ccec" (UID: "dbf4660e-4ff1-45b7-8ddb-df867099ccec"). InnerVolumeSpecName "kube-api-access-bz77b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:06.906557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.906501    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz77b\" (UniqueName: \"kubernetes.io/projected/dbf4660e-4ff1-45b7-8ddb-df867099ccec-kube-api-access-bz77b\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:06.906557 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:06.906526    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf4660e-4ff1-45b7-8ddb-df867099ccec-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:07.630352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:07.630327    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"
Oct 30 20:53:07.630352 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:07.630339    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc" event={"ID":"dbf4660e-4ff1-45b7-8ddb-df867099ccec","Type":"ContainerDied","Data":"3ff8344593dde3c5d1d0b1249fff8b2f86b8179bcbfbd4561ee97684dae27b42"}
Oct 30 20:53:07.630644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:07.630385    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff8344593dde3c5d1d0b1249fff8b2f86b8179bcbfbd4561ee97684dae27b42"
Oct 30 20:53:23.215312 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.215278    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"]
Oct 30 20:53:23.215948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.215934    5967 cpu_manager.go:414] "containerMap: removing container" podUID="dbf4660e-4ff1-45b7-8ddb-df867099ccec" containerName="revoke-keys"
Oct 30 20:53:23.216010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.215949    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf4660e-4ff1-45b7-8ddb-df867099ccec" containerName="revoke-keys"
Oct 30 20:53:23.216068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.216060    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf4660e-4ff1-45b7-8ddb-df867099ccec" containerName="revoke-keys"
Oct 30 20:53:23.221025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.221006    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.273982 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.273959    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:23.323151 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.323126    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:53:23.323292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.323176    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:23.323292 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.323210    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:23.327079 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.327060    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"]
Oct 30 20:53:23.370573 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.370548    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmwk\" (UniqueName: \"kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.370669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.370593    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.370878 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.370858    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.472042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.471971    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.472175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.472096    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmwk\" (UniqueName: \"kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.472175 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.472149    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.472831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.472811    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.474359 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.474338    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.533554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.533531    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmwk\" (UniqueName: \"kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk\") pod \"maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.534332 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.534312    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:23.915409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:23.915378    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"]
Oct 30 20:53:23.915581 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:53:23.915560    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e36eb9a_190e_42e0_940c_814062aa4039.slice/crio-1f9db8b696f0b5a0080bd3a8ff8c27cf378d71752a392548e0f49e6ddc02d1cd WatchSource:0}: Error finding container 1f9db8b696f0b5a0080bd3a8ff8c27cf378d71752a392548e0f49e6ddc02d1cd: Status 404 returned error can't find the container with id 1f9db8b696f0b5a0080bd3a8ff8c27cf378d71752a392548e0f49e6ddc02d1cd
Oct 30 20:53:24.815132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:24.815103    5967 generic.go:356] "Generic (PLEG): container finished" podID="2e36eb9a-190e-42e0-940c-814062aa4039" containerID="25194597f34dc60fa112c16cb6a92c382c68b7cef874e9a7d97aef3cabc60c8a" exitCode=0
Oct 30 20:53:24.815519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:24.815169    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5" event={"ID":"2e36eb9a-190e-42e0-940c-814062aa4039","Type":"ContainerDied","Data":"25194597f34dc60fa112c16cb6a92c382c68b7cef874e9a7d97aef3cabc60c8a"}
Oct 30 20:53:24.815519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:24.815217    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5" event={"ID":"2e36eb9a-190e-42e0-940c-814062aa4039","Type":"ContainerStarted","Data":"1f9db8b696f0b5a0080bd3a8ff8c27cf378d71752a392548e0f49e6ddc02d1cd"}
Oct 30 20:53:26.009611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.009588    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:26.012950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.012929    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk\" (UniqueName: \"kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk\") pod \"2e36eb9a-190e-42e0-940c-814062aa4039\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") "
Oct 30 20:53:26.013077 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.012981    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca\") pod \"2e36eb9a-190e-42e0-940c-814062aa4039\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") "
Oct 30 20:53:26.013142 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.013134    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca\") pod \"2e36eb9a-190e-42e0-940c-814062aa4039\" (UID: \"2e36eb9a-190e-42e0-940c-814062aa4039\") "
Oct 30 20:53:26.013355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.013331    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca" pod "2e36eb9a-190e-42e0-940c-814062aa4039" (UID: "2e36eb9a-190e-42e0-940c-814062aa4039"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:26.013578 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.013565    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:26.015109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.015090    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca" pod "2e36eb9a-190e-42e0-940c-814062aa4039" (UID: "2e36eb9a-190e-42e0-940c-814062aa4039"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:26.015330 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.015300    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk" pod "2e36eb9a-190e-42e0-940c-814062aa4039" (UID: "2e36eb9a-190e-42e0-940c-814062aa4039"). InnerVolumeSpecName "kube-api-access-pgmwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:26.115001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.114933    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e36eb9a-190e-42e0-940c-814062aa4039-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:26.115001 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.114956    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgmwk\" (UniqueName: \"kubernetes.io/projected/2e36eb9a-190e-42e0-940c-814062aa4039-kube-api-access-pgmwk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:26.834064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.834038    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"
Oct 30 20:53:26.834245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.834077    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5" event={"ID":"2e36eb9a-190e-42e0-940c-814062aa4039","Type":"ContainerDied","Data":"1f9db8b696f0b5a0080bd3a8ff8c27cf378d71752a392548e0f49e6ddc02d1cd"}
Oct 30 20:53:26.834245 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:26.834130    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f9db8b696f0b5a0080bd3a8ff8c27cf378d71752a392548e0f49e6ddc02d1cd"
Oct 30 20:53:31.981316 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:53:31.981246    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:53:34.047607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:34.047577    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"]
Oct 30 20:53:34.096816 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:34.096788    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33007785-vhz4h"]
Oct 30 20:53:35.267275 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.267242    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"]
Oct 30 20:53:35.267932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.267918    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2e36eb9a-190e-42e0-940c-814062aa4039" containerName="revoke-keys"
Oct 30 20:53:35.267995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.267933    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e36eb9a-190e-42e0-940c-814062aa4039" containerName="revoke-keys"
Oct 30 20:53:35.268090 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.268081    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e36eb9a-190e-42e0-940c-814062aa4039" containerName="revoke-keys"
Oct 30 20:53:35.271907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.271890    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.338798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.338721    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.338942 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.338847    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2xg\" (UniqueName: \"kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.339017 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.338983    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.373805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.373350    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:35.373805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.373400    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:35.373805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.373502    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:53:35.373805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.373577    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:35.378692 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.378667    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"]
Oct 30 20:53:35.440546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.440518    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.440697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.440565    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2xg\" (UniqueName: \"kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.440697 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.440630    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.441288 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.441272    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.442984 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.442963    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.497473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.497444    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2xg\" (UniqueName: \"kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.586851 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.584850    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04824aa9-8a45-4ddb-aafd-b0a186e28062" path="/var/lib/kubelet/pods/04824aa9-8a45-4ddb-aafd-b0a186e28062/volumes"
Oct 30 20:53:35.589531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.588940    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:35.977646 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:35.977618    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"]
Oct 30 20:53:35.978278 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:53:35.978246    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccc9c94_3c3f_4b35_aefe_36a51acd4773.slice/crio-e93ac7692cb5ce0d085f3f17f32c6cff15fa22d0c62997a1b620a5c3a40573c6 WatchSource:0}: Error finding container e93ac7692cb5ce0d085f3f17f32c6cff15fa22d0c62997a1b620a5c3a40573c6: Status 404 returned error can't find the container with id e93ac7692cb5ce0d085f3f17f32c6cff15fa22d0c62997a1b620a5c3a40573c6
Oct 30 20:53:36.966607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:36.966516    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g" event={"ID":"2ccc9c94-3c3f-4b35-aefe-36a51acd4773","Type":"ContainerStarted","Data":"b0a523d1030f4e4a33a012d738d19791ecdb90e4af06175fcd14c351dd64d1d7"}
Oct 30 20:53:36.966607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:36.966576    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g" event={"ID":"2ccc9c94-3c3f-4b35-aefe-36a51acd4773","Type":"ContainerStarted","Data":"e93ac7692cb5ce0d085f3f17f32c6cff15fa22d0c62997a1b620a5c3a40573c6"}
Oct 30 20:53:37.082383 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:37.082330    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g" podStartSLOduration=2.082316971 podStartE2EDuration="2.082316971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:37.080724583 +0000 UTC m=+1997.681526535" watchObservedRunningTime="2026-10-30 20:53:37.082316971 +0000 UTC m=+1997.683118931"
Oct 30 20:53:37.763356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:37.763323    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"]
Oct 30 20:53:37.766528 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:37.766502    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0ac4e5-b6l4w"]
Oct 30 20:53:37.987200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:37.987163    5967 generic.go:356] "Generic (PLEG): container finished" podID="2ccc9c94-3c3f-4b35-aefe-36a51acd4773" containerID="b0a523d1030f4e4a33a012d738d19791ecdb90e4af06175fcd14c351dd64d1d7" exitCode=0
Oct 30 20:53:37.987868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:37.987839    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g" event={"ID":"2ccc9c94-3c3f-4b35-aefe-36a51acd4773","Type":"ContainerDied","Data":"b0a523d1030f4e4a33a012d738d19791ecdb90e4af06175fcd14c351dd64d1d7"}
Oct 30 20:53:39.163360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.163336    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:39.246063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.246036    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca\") pod \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") "
Oct 30 20:53:39.246226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.246075    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg\" (UniqueName: \"kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg\") pod \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") "
Oct 30 20:53:39.246226 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.246135    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca\") pod \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\" (UID: \"2ccc9c94-3c3f-4b35-aefe-36a51acd4773\") "
Oct 30 20:53:39.246413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.246394    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca" pod "2ccc9c94-3c3f-4b35-aefe-36a51acd4773" (UID: "2ccc9c94-3c3f-4b35-aefe-36a51acd4773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:39.246981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.246934    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:39.248372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.248347    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca" pod "2ccc9c94-3c3f-4b35-aefe-36a51acd4773" (UID: "2ccc9c94-3c3f-4b35-aefe-36a51acd4773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:39.248474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.248428    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg" pod "2ccc9c94-3c3f-4b35-aefe-36a51acd4773" (UID: "2ccc9c94-3c3f-4b35-aefe-36a51acd4773"). InnerVolumeSpecName "kube-api-access-cg2xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:39.347986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.347960    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2xg\" (UniqueName: \"kubernetes.io/projected/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-kube-api-access-cg2xg\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:39.347986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.347985    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ccc9c94-3c3f-4b35-aefe-36a51acd4773-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:39.543572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:39.543500    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d98592-cf77-413f-8a64-f044af899f58" path="/var/lib/kubelet/pods/84d98592-cf77-413f-8a64-f044af899f58/volumes"
Oct 30 20:53:40.004673 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:40.004648    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"
Oct 30 20:53:40.004883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:40.004691    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g" event={"ID":"2ccc9c94-3c3f-4b35-aefe-36a51acd4773","Type":"ContainerDied","Data":"e93ac7692cb5ce0d085f3f17f32c6cff15fa22d0c62997a1b620a5c3a40573c6"}
Oct 30 20:53:40.004883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:40.004766    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e93ac7692cb5ce0d085f3f17f32c6cff15fa22d0c62997a1b620a5c3a40573c6"
Oct 30 20:53:44.203622 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:44.203588    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"]
Oct 30 20:53:44.252390 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:44.252364    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-99c030-frbq2"]
Oct 30 20:53:45.580806 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:45.580767    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d987ceb1-fbe3-435c-99a1-211124844bbc" path="/var/lib/kubelet/pods/d987ceb1-fbe3-435c-99a1-211124844bbc/volumes"
Oct 30 20:53:46.217331 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.217294    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"]
Oct 30 20:53:46.217988 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.217975    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2ccc9c94-3c3f-4b35-aefe-36a51acd4773" containerName="revoke-keys"
Oct 30 20:53:46.218038 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.217990    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ccc9c94-3c3f-4b35-aefe-36a51acd4773" containerName="revoke-keys"
Oct 30 20:53:46.218134 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.218124    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ccc9c94-3c3f-4b35-aefe-36a51acd4773" containerName="revoke-keys"
Oct 30 20:53:46.221688 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.221669    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.275481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.275458    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.275614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.275585    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhzt\" (UniqueName: \"kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.275664 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.275643    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.275977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.275960    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:46.276126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.276109    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:46.325606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.325574    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:46.325761 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.325624    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:53:46.331268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.331243    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"]
Oct 30 20:53:46.377101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.377080    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.377205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.377191    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhzt\" (UniqueName: \"kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.377268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.377257    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.377850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.377831    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.379580 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.379561    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.434432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.434410    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhzt\" (UniqueName: \"kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.535284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.535219    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:46.918491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:46.918460    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"]
Oct 30 20:53:46.918996 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:53:46.918975    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405b067f_65ff_4a61_8c33_80dbd82eb590.slice/crio-5c2d354e5c3f0c5d88afa85c827df0622285f5e610f53d208a4d96bd19dc1aa8 WatchSource:0}: Error finding container 5c2d354e5c3f0c5d88afa85c827df0622285f5e610f53d208a4d96bd19dc1aa8: Status 404 returned error can't find the container with id 5c2d354e5c3f0c5d88afa85c827df0622285f5e610f53d208a4d96bd19dc1aa8
Oct 30 20:53:47.096376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:47.096350    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27" event={"ID":"405b067f-65ff-4a61-8c33-80dbd82eb590","Type":"ContainerStarted","Data":"5c2d354e5c3f0c5d88afa85c827df0622285f5e610f53d208a4d96bd19dc1aa8"}
Oct 30 20:53:48.106115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:48.106039    5967 generic.go:356] "Generic (PLEG): container finished" podID="405b067f-65ff-4a61-8c33-80dbd82eb590" containerID="264e69ca5ac4a0ecf20cf814dec8c1da2765b83695b1b2e430432a384d72e920" exitCode=0
Oct 30 20:53:48.106469 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:48.106107    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27" event={"ID":"405b067f-65ff-4a61-8c33-80dbd82eb590","Type":"ContainerDied","Data":"264e69ca5ac4a0ecf20cf814dec8c1da2765b83695b1b2e430432a384d72e920"}
Oct 30 20:53:49.265046 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.265025    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:49.334132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.334103    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca\") pod \"405b067f-65ff-4a61-8c33-80dbd82eb590\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") "
Oct 30 20:53:49.334304 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.334145    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt\" (UniqueName: \"kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt\") pod \"405b067f-65ff-4a61-8c33-80dbd82eb590\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") "
Oct 30 20:53:49.334386 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.334329    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca\") pod \"405b067f-65ff-4a61-8c33-80dbd82eb590\" (UID: \"405b067f-65ff-4a61-8c33-80dbd82eb590\") "
Oct 30 20:53:49.334698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.334668    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca" pod "405b067f-65ff-4a61-8c33-80dbd82eb590" (UID: "405b067f-65ff-4a61-8c33-80dbd82eb590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:49.334924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.334898    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:49.336357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.336336    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt" pod "405b067f-65ff-4a61-8c33-80dbd82eb590" (UID: "405b067f-65ff-4a61-8c33-80dbd82eb590"). InnerVolumeSpecName "kube-api-access-qbhzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:49.336455 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.336355    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca" pod "405b067f-65ff-4a61-8c33-80dbd82eb590" (UID: "405b067f-65ff-4a61-8c33-80dbd82eb590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:49.441690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.437129    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405b067f-65ff-4a61-8c33-80dbd82eb590-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:49.441690 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:49.437165    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhzt\" (UniqueName: \"kubernetes.io/projected/405b067f-65ff-4a61-8c33-80dbd82eb590-kube-api-access-qbhzt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:50.184762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:50.184568    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27" event={"ID":"405b067f-65ff-4a61-8c33-80dbd82eb590","Type":"ContainerDied","Data":"5c2d354e5c3f0c5d88afa85c827df0622285f5e610f53d208a4d96bd19dc1aa8"}
Oct 30 20:53:50.184762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:50.184637    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c2d354e5c3f0c5d88afa85c827df0622285f5e610f53d208a4d96bd19dc1aa8"
Oct 30 20:53:50.184762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:50.184711    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"
Oct 30 20:53:52.249830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.249797    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"]
Oct 30 20:53:52.250733 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.250714    5967 cpu_manager.go:414] "containerMap: removing container" podUID="405b067f-65ff-4a61-8c33-80dbd82eb590" containerName="revoke-keys"
Oct 30 20:53:52.250820 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.250752    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405b067f-65ff-4a61-8c33-80dbd82eb590" containerName="revoke-keys"
Oct 30 20:53:52.250888 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.250878    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="405b067f-65ff-4a61-8c33-80dbd82eb590" containerName="revoke-keys"
Oct 30 20:53:52.254003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.253986    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.300356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.300329    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.300480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.300379    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.300480 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.300400    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9l2\" (UniqueName: \"kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.358835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.358773    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:52.358835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.358789    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:52.358835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.358776    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:52.358835 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.358821    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:53:52.362923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.362902    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"]
Oct 30 20:53:52.401687 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.401660    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.401822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.401711    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.401822 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.401733    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9l2\" (UniqueName: \"kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.402402 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.402380    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.404094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.404077    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.461102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.461075    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9l2\" (UniqueName: \"kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.567174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.567123    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:52.968897 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:52.968871    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"]
Oct 30 20:53:52.970450 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:53:52.970427    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8877f4ed_d319_42f6_976e_928c229f6008.slice/crio-3254250a8dd46b5d38a99eb38d14611224ef87b7cae4dfa507ecc3a52bbecbe6 WatchSource:0}: Error finding container 3254250a8dd46b5d38a99eb38d14611224ef87b7cae4dfa507ecc3a52bbecbe6: Status 404 returned error can't find the container with id 3254250a8dd46b5d38a99eb38d14611224ef87b7cae4dfa507ecc3a52bbecbe6
Oct 30 20:53:53.218460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:53.218372    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr" event={"ID":"8877f4ed-d319-42f6-976e-928c229f6008","Type":"ContainerStarted","Data":"3254250a8dd46b5d38a99eb38d14611224ef87b7cae4dfa507ecc3a52bbecbe6"}
Oct 30 20:53:54.226907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:54.226874    5967 generic.go:356] "Generic (PLEG): container finished" podID="8877f4ed-d319-42f6-976e-928c229f6008" containerID="be13c20cfefc05e897347710c238557354c5b1dcbcfa96ee11148d9300590876" exitCode=0
Oct 30 20:53:54.226907 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:54.226909    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr" event={"ID":"8877f4ed-d319-42f6-976e-928c229f6008","Type":"ContainerDied","Data":"be13c20cfefc05e897347710c238557354c5b1dcbcfa96ee11148d9300590876"}
Oct 30 20:53:55.433562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.433533    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:55.563380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.563298    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca\") pod \"8877f4ed-d319-42f6-976e-928c229f6008\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") "
Oct 30 20:53:55.563542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.563410    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2\" (UniqueName: \"kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2\") pod \"8877f4ed-d319-42f6-976e-928c229f6008\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") "
Oct 30 20:53:55.563542 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.563462    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca\") pod \"8877f4ed-d319-42f6-976e-928c229f6008\" (UID: \"8877f4ed-d319-42f6-976e-928c229f6008\") "
Oct 30 20:53:55.564112 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.564033    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca" pod "8877f4ed-d319-42f6-976e-928c229f6008" (UID: "8877f4ed-d319-42f6-976e-928c229f6008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:55.564300 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.564241    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:55.565672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.565643    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca" pod "8877f4ed-d319-42f6-976e-928c229f6008" (UID: "8877f4ed-d319-42f6-976e-928c229f6008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:55.566272 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.566247    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2" pod "8877f4ed-d319-42f6-976e-928c229f6008" (UID: "8877f4ed-d319-42f6-976e-928c229f6008"). InnerVolumeSpecName "kube-api-access-jq9l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:55.665417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.665384    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8877f4ed-d319-42f6-976e-928c229f6008-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:55.665417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:55.665419    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9l2\" (UniqueName: \"kubernetes.io/projected/8877f4ed-d319-42f6-976e-928c229f6008-kube-api-access-jq9l2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.136853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:56.136824    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"]
Oct 30 20:53:56.141915 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:56.141891    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-68cgb"]
Oct 30 20:53:56.263885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:56.263853    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"
Oct 30 20:53:56.263885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:56.263867    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr" event={"ID":"8877f4ed-d319-42f6-976e-928c229f6008","Type":"ContainerDied","Data":"3254250a8dd46b5d38a99eb38d14611224ef87b7cae4dfa507ecc3a52bbecbe6"}
Oct 30 20:53:56.264097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:56.263921    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3254250a8dd46b5d38a99eb38d14611224ef87b7cae4dfa507ecc3a52bbecbe6"
Oct 30 20:53:57.544310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:53:57.544272    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae3d458-9a41-4e87-9044-ac0cb1a99a24" path="/var/lib/kubelet/pods/5ae3d458-9a41-4e87-9044-ac0cb1a99a24/volumes"
Oct 30 20:54:02.002597 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:02.002558    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:54:07.491479 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:07.491448    5967 generic.go:356] "Generic (PLEG): container finished" podID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerID="8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94" exitCode=0
Oct 30 20:54:07.491977 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:07.491525    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" event={"ID":"ce9a7b15-288a-4b84-ac8f-b179c382dc9e","Type":"ContainerDied","Data":"8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"}
Oct 30 20:54:07.492027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:07.492005    5967 scope.go:122] "RemoveContainer" containerID="8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"
Oct 30 20:54:07.493582 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:07.493551    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:07.493669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:07.493619    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:07.493669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:07.493650    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:08.526203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:08.526160    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" event={"ID":"ce9a7b15-288a-4b84-ac8f-b179c382dc9e","Type":"ContainerStarted","Data":"98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59"}
Oct 30 20:54:08.617629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:08.617590    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:54:08.619187 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:08.619163    5967 patch_prober.go:29] interesting pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 20:54:08.619271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:08.619212    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 20:54:09.618033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:09.617998    5967 patch_prober.go:29] interesting pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 20:54:09.618514 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:09.618055    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 20:54:10.617768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:10.617706    5967 patch_prober.go:29] interesting pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 20:54:10.617960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:10.617785    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 20:54:10.802023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:10.801991    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:54:11.617844 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:11.617809    5967 patch_prober.go:29] interesting pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 20:54:11.618028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:11.617866    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 20:54:12.618256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:12.618222    5967 patch_prober.go:29] interesting pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 20:54:12.618711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:12.618275    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 20:54:13.617250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:13.617213    5967 patch_prober.go:29] interesting pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 20:54:13.617418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:13.617287    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 20:54:14.622413 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:14.622387    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:54:14.623362 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:14.623343    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:54:27.720356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:27.720316    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"]
Oct 30 20:54:27.768987 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:27.768959    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3d13a-4f9hc"]
Oct 30 20:54:29.543845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:29.543816    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf4660e-4ff1-45b7-8ddb-df867099ccec" path="/var/lib/kubelet/pods/dbf4660e-4ff1-45b7-8ddb-df867099ccec/volumes"
Oct 30 20:54:31.981136 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:31.981098    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:54:34.883602 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:34.883562    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"]
Oct 30 20:54:34.884100 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:34.883930    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy" containerID="cri-o://97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa" gracePeriod=30
Oct 30 20:54:39.939125 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:39.939099    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c1c1e0_90bb_45fa_b30b_d9d235d502b2.slice/crio-97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:54:40.169213 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.169158    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:54:40.241160 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241133    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241176    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241196    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241225    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241359    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241434    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:40.241507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241482    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241512    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241555    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241579    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token\") pod \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\" (UID: \"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2\") "
Oct 30 20:54:40.241672 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241626    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:40.241926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241830    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:40.241926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.241858    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:40.242199 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.242077    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:40.242537 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.242512    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.242603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.242542    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.242603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.242557    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.242603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.242571    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.242603 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.242584    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.243689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.243656    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "kube-api-access-hfstq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:40.243848 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.243715    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:40.243917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.243867    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:40.243917 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.243897    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token" pod "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" (UID: "a9c1c1e0-90bb-45fa-b30b-d9d235d502b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:40.344503 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.344472    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.344663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.344507    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.344663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.344521    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfstq\" (UniqueName: \"kubernetes.io/projected/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-kube-api-access-hfstq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.344663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.344536    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:40.899080 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.899050    5967 generic.go:356] "Generic (PLEG): container finished" podID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerID="97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa" exitCode=0
Oct 30 20:54:40.899271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.899100    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"
Oct 30 20:54:40.899271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.899134    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" event={"ID":"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2","Type":"ContainerDied","Data":"97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa"}
Oct 30 20:54:40.899271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.899190    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j" event={"ID":"a9c1c1e0-90bb-45fa-b30b-d9d235d502b2","Type":"ContainerDied","Data":"45d4b9ac72c0ad020deda019ee543daa95ab1fa3a98505775c1853ab5dd2d26e"}
Oct 30 20:54:40.899271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.899217    5967 scope.go:122] "RemoveContainer" containerID="97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa"
Oct 30 20:54:40.910544 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.910528    5967 scope.go:122] "RemoveContainer" containerID="b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"
Oct 30 20:54:40.919889 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.919861    5967 scope.go:122] "RemoveContainer" containerID="97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa"
Oct 30 20:54:40.920110 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:40.920095    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa\": container with ID starting with 97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa not found: ID does not exist" containerID="97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa"
Oct 30 20:54:40.920145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.920116    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa"} err="failed to get container status \"97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa\": rpc error: code = NotFound desc = could not find container \"97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa\": container with ID starting with 97d113bf421e717221f545c995f0418b6ab3793058abcdde1f0d38521c174dfa not found: ID does not exist"
Oct 30 20:54:40.920145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.920134    5967 scope.go:122] "RemoveContainer" containerID="b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"
Oct 30 20:54:40.920321 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:40.920305    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6\": container with ID starting with b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6 not found: ID does not exist" containerID="b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"
Oct 30 20:54:40.920357 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:40.920324    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6"} err="failed to get container status \"b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6\": rpc error: code = NotFound desc = could not find container \"b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6\": container with ID starting with b7518b5eadfddf0bcb857c88f3b020e93604897e3364009ca9c0d5e2565801f6 not found: ID does not exist"
Oct 30 20:54:41.064927 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:41.064902    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"]
Oct 30 20:54:41.114504 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:41.114462    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j"]
Oct 30 20:54:41.543519 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:41.543485    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" path="/var/lib/kubelet/pods/a9c1c1e0-90bb-45fa-b30b-d9d235d502b2/volumes"
Oct 30 20:54:43.776751 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:43.776691    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"]
Oct 30 20:54:43.829765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:43.825968    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-034eba-8p95g"]
Oct 30 20:54:45.543320 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:45.543288    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ccc9c94-3c3f-4b35-aefe-36a51acd4773" path="/var/lib/kubelet/pods/2ccc9c94-3c3f-4b35-aefe-36a51acd4773/volumes"
Oct 30 20:54:46.002265 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:46.002226    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"]
Oct 30 20:54:46.033329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:46.029716    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f09ec-vxwz5"]
Oct 30 20:54:47.545374 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:47.545343    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e36eb9a-190e-42e0-940c-814062aa4039" path="/var/lib/kubelet/pods/2e36eb9a-190e-42e0-940c-814062aa4039/volumes"
Oct 30 20:54:50.591113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:50.591079    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"]
Oct 30 20:54:50.591535 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:50.591297    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy" containerID="cri-o://c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d" gracePeriod=30
Oct 30 20:54:51.416954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.416921    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"]
Oct 30 20:54:51.417574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417560    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy"
Oct 30 20:54:51.417619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417576    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy"
Oct 30 20:54:51.417619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417586    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8877f4ed-d319-42f6-976e-928c229f6008" containerName="revoke-keys"
Oct 30 20:54:51.417619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417591    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8877f4ed-d319-42f6-976e-928c229f6008" containerName="revoke-keys"
Oct 30 20:54:51.417619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417609    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy"
Oct 30 20:54:51.417619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417614    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy"
Oct 30 20:54:51.417791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417693    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy"
Oct 30 20:54:51.417791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417700    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8877f4ed-d319-42f6-976e-928c229f6008" containerName="revoke-keys"
Oct 30 20:54:51.417791 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.417710    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c1c1e0-90bb-45fa-b30b-d9d235d502b2" containerName="istio-proxy"
Oct 30 20:54:51.422566 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.422550    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.480831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.480805    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.480971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.480951    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qngv\" (UniqueName: \"kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.481126 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.481110    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.526900 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.526868    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:51.526900 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.526869    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:51.527102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.526956    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:51.527102 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.527088    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:54:51.530654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.530633    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"]
Oct 30 20:54:51.582663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.582642    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.582819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.582717    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qngv\" (UniqueName: \"kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.582892 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.582860    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.583280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.583258    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.585366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.585345    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.639764 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.639726    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qngv\" (UniqueName: \"kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:51.736268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:51.736240    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:52.156021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:52.155994    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"]
Oct 30 20:54:52.156708 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:54:52.156684    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4b1aa9_df57_47b7_8511_ecc36ff851ba.slice/crio-dd211af8e1a7c6f2b287095b31a0817ff953391c872f2658cdbced17f090a7b7 WatchSource:0}: Error finding container dd211af8e1a7c6f2b287095b31a0817ff953391c872f2658cdbced17f090a7b7: Status 404 returned error can't find the container with id dd211af8e1a7c6f2b287095b31a0817ff953391c872f2658cdbced17f090a7b7
Oct 30 20:54:53.113166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:53.113083    5967 generic.go:356] "Generic (PLEG): container finished" podID="2b4b1aa9-df57-47b7-8511-ecc36ff851ba" containerID="977cc3c59979b9d8821931dca5f8a5ffe43ad94d8e0d39e79c18b7dfd820b269" exitCode=0
Oct 30 20:54:53.113166 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:53.113144    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6" event={"ID":"2b4b1aa9-df57-47b7-8511-ecc36ff851ba","Type":"ContainerDied","Data":"977cc3c59979b9d8821931dca5f8a5ffe43ad94d8e0d39e79c18b7dfd820b269"}
Oct 30 20:54:53.113722 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:53.113191    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6" event={"ID":"2b4b1aa9-df57-47b7-8511-ecc36ff851ba","Type":"ContainerStarted","Data":"dd211af8e1a7c6f2b287095b31a0817ff953391c872f2658cdbced17f090a7b7"}
Oct 30 20:54:54.302781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.297303    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:54.345782 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.345711    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"]
Oct 30 20:54:54.345967 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.345919    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy" containerID="cri-o://fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37" gracePeriod=30
Oct 30 20:54:54.363923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.363890    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca\") pod \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") "
Oct 30 20:54:54.364127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.364110    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca\") pod \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") "
Oct 30 20:54:54.364303 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.364275    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca" pod "2b4b1aa9-df57-47b7-8511-ecc36ff851ba" (UID: "2b4b1aa9-df57-47b7-8511-ecc36ff851ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:54.364498 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.364471    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv\" (UniqueName: \"kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv\") pod \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\" (UID: \"2b4b1aa9-df57-47b7-8511-ecc36ff851ba\") "
Oct 30 20:54:54.365438 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.365416    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:54.366587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.366560    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca" pod "2b4b1aa9-df57-47b7-8511-ecc36ff851ba" (UID: "2b4b1aa9-df57-47b7-8511-ecc36ff851ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:54.366836 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.366816    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv" pod "2b4b1aa9-df57-47b7-8511-ecc36ff851ba" (UID: "2b4b1aa9-df57-47b7-8511-ecc36ff851ba"). InnerVolumeSpecName "kube-api-access-6qngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:54.529762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.529043    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qngv\" (UniqueName: \"kubernetes.io/projected/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-kube-api-access-6qngv\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:54.529762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:54.529084    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4b1aa9-df57-47b7-8511-ecc36ff851ba-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:55.128652 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:55.128624    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"
Oct 30 20:54:55.128883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:55.128653    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6" event={"ID":"2b4b1aa9-df57-47b7-8511-ecc36ff851ba","Type":"ContainerDied","Data":"dd211af8e1a7c6f2b287095b31a0817ff953391c872f2658cdbced17f090a7b7"}
Oct 30 20:54:55.128883 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:55.128699    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd211af8e1a7c6f2b287095b31a0817ff953391c872f2658cdbced17f090a7b7"
Oct 30 20:54:55.920904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:55.920882    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:54:56.058569 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058544    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.058728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058621    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.058728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058711    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.058843 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058768    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.058843 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058801    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.058843 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058831    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.058996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058852    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:56.058996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058964    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:56.058996 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.058970    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.059165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.059066    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:56.059165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.059083    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.059165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.059127    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo\") pod \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\" (UID: \"9dbc28a0-6c60-40a2-ac35-be0f154aea4b\") "
Oct 30 20:54:56.059165 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.059149    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:56.059377 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.059211    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:56.060098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.060073    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.060214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.060103    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.060214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.060118    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.060214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.060132    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.060214 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.060149    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.061145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.061118    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "kube-api-access-z5mms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:56.061230 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.061115    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:56.061230 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.061201    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:56.061422 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.061405    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo" pod "9dbc28a0-6c60-40a2-ac35-be0f154aea4b" (UID: "9dbc28a0-6c60-40a2-ac35-be0f154aea4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:56.138052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.137976    5967 generic.go:356] "Generic (PLEG): container finished" podID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerID="c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d" exitCode=0
Oct 30 20:54:56.138052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.138034    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"
Oct 30 20:54:56.138253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.138058    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" event={"ID":"9dbc28a0-6c60-40a2-ac35-be0f154aea4b","Type":"ContainerDied","Data":"c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d"}
Oct 30 20:54:56.138253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.138117    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s" event={"ID":"9dbc28a0-6c60-40a2-ac35-be0f154aea4b","Type":"ContainerDied","Data":"2d5018da9a00862549828ee5ce2a17c12d6d283f09b65cdff6051f454d7abbba"}
Oct 30 20:54:56.138253 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.138140    5967 scope.go:122] "RemoveContainer" containerID="c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d"
Oct 30 20:54:56.150307 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.150292    5967 scope.go:122] "RemoveContainer" containerID="d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"
Oct 30 20:54:56.160091 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.160073    5967 scope.go:122] "RemoveContainer" containerID="c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d"
Oct 30 20:54:56.160370 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:56.160346    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d\": container with ID starting with c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d not found: ID does not exist" containerID="c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d"
Oct 30 20:54:56.160449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.160376    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d"} err="failed to get container status \"c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d\": rpc error: code = NotFound desc = could not find container \"c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d\": container with ID starting with c5786540cb2b1e62fb3ea461d114e29a2c3b6c4617896ac8d362e214e26bd31d not found: ID does not exist"
Oct 30 20:54:56.160449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.160394    5967 scope.go:122] "RemoveContainer" containerID="d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"
Oct 30 20:54:56.160625 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:54:56.160612    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665\": container with ID starting with d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665 not found: ID does not exist" containerID="d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"
Oct 30 20:54:56.160678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.160629    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665"} err="failed to get container status \"d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665\": rpc error: code = NotFound desc = could not find container \"d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665\": container with ID starting with d6235cf86b8ff551a487c2548533e3104c9fcf02a4a1650dde3ace030f16e665 not found: ID does not exist"
Oct 30 20:54:56.161236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.161214    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.161326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.161240    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.161326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.161255    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-kube-api-access-z5mms\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.161326 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.161269    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbc28a0-6c60-40a2-ac35-be0f154aea4b-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:56.303217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.303185    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"]
Oct 30 20:54:56.309576 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.309552    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s"]
Oct 30 20:54:56.591251 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591223    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"]
Oct 30 20:54:56.591862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591849    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy"
Oct 30 20:54:56.591914 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591864    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy"
Oct 30 20:54:56.591914 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591876    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2b4b1aa9-df57-47b7-8511-ecc36ff851ba" containerName="revoke-keys"
Oct 30 20:54:56.591914 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591880    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4b1aa9-df57-47b7-8511-ecc36ff851ba" containerName="revoke-keys"
Oct 30 20:54:56.592006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591929    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy"
Oct 30 20:54:56.592006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.591935    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy"
Oct 30 20:54:56.592068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.592019    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy"
Oct 30 20:54:56.592068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.592026    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4b1aa9-df57-47b7-8511-ecc36ff851ba" containerName="revoke-keys"
Oct 30 20:54:56.592068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.592039    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" containerName="istio-proxy"
Oct 30 20:54:56.596895 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.596880    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.649540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.649512    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:56.649775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.649657    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:56.649775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.649713    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:56.649960 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.649782    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:54:56.651773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.651735    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"]
Oct 30 20:54:56.774395 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.774366    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4c2\" (UniqueName: \"kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.774562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.774406    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.774562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.774428    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.876381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.876306    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4c2\" (UniqueName: \"kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.876381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.876345    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.876381 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.876370    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.877094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.877075    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.878809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.878790    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:56.932511 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:56.932490    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4c2\" (UniqueName: \"kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:57.219018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:57.218988    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:54:57.544081 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:57.544010    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbc28a0-6c60-40a2-ac35-be0f154aea4b" path="/var/lib/kubelet/pods/9dbc28a0-6c60-40a2-ac35-be0f154aea4b/volumes"
Oct 30 20:54:57.597006 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:57.596977    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"]
Oct 30 20:54:57.597425 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:54:57.597402    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3a2612_d065_4443_a969_c8c851af345d.slice/crio-080cfedf172e074298520b06146e1344dff07ef57be54524427ffbec6ea6a50f WatchSource:0}: Error finding container 080cfedf172e074298520b06146e1344dff07ef57be54524427ffbec6ea6a50f: Status 404 returned error can't find the container with id 080cfedf172e074298520b06146e1344dff07ef57be54524427ffbec6ea6a50f
Oct 30 20:54:58.155973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:58.155878    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf" event={"ID":"3d3a2612-d065-4443-a969-c8c851af345d","Type":"ContainerStarted","Data":"5f68d020bd72ba188cde89104a3c042a420627db2f2573b4b5729aa016b3ae7e"}
Oct 30 20:54:58.155973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:58.155941    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf" event={"ID":"3d3a2612-d065-4443-a969-c8c851af345d","Type":"ContainerStarted","Data":"080cfedf172e074298520b06146e1344dff07ef57be54524427ffbec6ea6a50f"}
Oct 30 20:54:58.271962 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:58.271907    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf" podStartSLOduration=2.271895102 podStartE2EDuration="2.271895102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:58.270039349 +0000 UTC m=+2078.870841309" watchObservedRunningTime="2026-10-30 20:54:58.271895102 +0000 UTC m=+2078.872697062"
Oct 30 20:54:59.165043 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.165013    5967 generic.go:356] "Generic (PLEG): container finished" podID="3d3a2612-d065-4443-a969-c8c851af345d" containerID="5f68d020bd72ba188cde89104a3c042a420627db2f2573b4b5729aa016b3ae7e" exitCode=0
Oct 30 20:54:59.165642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.165114    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf" event={"ID":"3d3a2612-d065-4443-a969-c8c851af345d","Type":"ContainerDied","Data":"5f68d020bd72ba188cde89104a3c042a420627db2f2573b4b5729aa016b3ae7e"}
Oct 30 20:54:59.634263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.634228    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:54:59.711847 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.711820    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.711991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.711915    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.711991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.711952    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.711991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.711974    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.712097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.711993    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.712097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712047    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.712097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712063    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:59.712097 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712075    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.712271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712185    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.712271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712221    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:59.712271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712242    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:59.712271 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712263    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket\") pod \"45f81c78-fe7a-4e77-afbb-fd21816db221\" (UID: \"45f81c78-fe7a-4e77-afbb-fd21816db221\") "
Oct 30 20:54:59.712484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712313    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:59.712541 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712511    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:59.712904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712829    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f81c78-fe7a-4e77-afbb-fd21816db221-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.712904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712849    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.712904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712859    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.712904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712872    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.712904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.712885    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.714481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.714454    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "kube-api-access-mb69b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:59.714481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.714454    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:59.714933 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.714919    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:59.715014 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.714939    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo" pod "45f81c78-fe7a-4e77-afbb-fd21816db221" (UID: "45f81c78-fe7a-4e77-afbb-fd21816db221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:59.814284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.814244    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.814284 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.814282    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb69b\" (UniqueName: \"kubernetes.io/projected/45f81c78-fe7a-4e77-afbb-fd21816db221-kube-api-access-mb69b\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.814513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.814299    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:59.814513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:54:59.814315    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f81c78-fe7a-4e77-afbb-fd21816db221-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:00.183417 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.183323    5967 generic.go:356] "Generic (PLEG): container finished" podID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerID="fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37" exitCode=0
Oct 30 20:55:00.183830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.183422    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" event={"ID":"45f81c78-fe7a-4e77-afbb-fd21816db221","Type":"ContainerDied","Data":"fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37"}
Oct 30 20:55:00.183830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.183434    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"
Oct 30 20:55:00.183830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.183688    5967 scope.go:122] "RemoveContainer" containerID="fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37"
Oct 30 20:55:00.183978 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.183827    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm" event={"ID":"45f81c78-fe7a-4e77-afbb-fd21816db221","Type":"ContainerDied","Data":"e4e31f67d5c6fd30ab4d0e968fd8d5e7c5a2e909fd67c393611a4167120c0b1d"}
Oct 30 20:55:00.199009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.198995    5967 scope.go:122] "RemoveContainer" containerID="de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"
Oct 30 20:55:00.212171 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.212147    5967 scope.go:122] "RemoveContainer" containerID="fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37"
Oct 30 20:55:00.212469 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:00.212447    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37\": container with ID starting with fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37 not found: ID does not exist" containerID="fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37"
Oct 30 20:55:00.212553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.212477    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37"} err="failed to get container status \"fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37\": rpc error: code = NotFound desc = could not find container \"fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37\": container with ID starting with fc604f22854483dd5b31016e6afd96c12b02f9a74444f2b61cad4872b8c77c37 not found: ID does not exist"
Oct 30 20:55:00.212553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.212494    5967 scope.go:122] "RemoveContainer" containerID="de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"
Oct 30 20:55:00.212807 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:00.212785    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f\": container with ID starting with de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f not found: ID does not exist" containerID="de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"
Oct 30 20:55:00.212893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.212809    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f"} err="failed to get container status \"de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f\": rpc error: code = NotFound desc = could not find container \"de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f\": container with ID starting with de791bbe2da7abdc9cbbc89778347656fcbabfeb778aade1f5200c8dedd7e63f not found: ID does not exist"
Oct 30 20:55:00.355258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.355224    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"]
Oct 30 20:55:00.368009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.367979    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:55:00.400627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.400598    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm"]
Oct 30 20:55:00.420075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.420050    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2\" (UniqueName: \"kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2\") pod \"3d3a2612-d065-4443-a969-c8c851af345d\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") "
Oct 30 20:55:00.420238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.420132    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca\") pod \"3d3a2612-d065-4443-a969-c8c851af345d\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") "
Oct 30 20:55:00.420304 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.420271    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca\") pod \"3d3a2612-d065-4443-a969-c8c851af345d\" (UID: \"3d3a2612-d065-4443-a969-c8c851af345d\") "
Oct 30 20:55:00.420540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.420513    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca" pod "3d3a2612-d065-4443-a969-c8c851af345d" (UID: "3d3a2612-d065-4443-a969-c8c851af345d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:00.420904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.420885    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:00.422404 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.422374    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca" pod "3d3a2612-d065-4443-a969-c8c851af345d" (UID: "3d3a2612-d065-4443-a969-c8c851af345d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:00.422497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.422422    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2" pod "3d3a2612-d065-4443-a969-c8c851af345d" (UID: "3d3a2612-d065-4443-a969-c8c851af345d"). InnerVolumeSpecName "kube-api-access-bl4c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:00.521753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.521704    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a2612-d065-4443-a969-c8c851af345d-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:00.521753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:00.521733    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4c2\" (UniqueName: \"kubernetes.io/projected/3d3a2612-d065-4443-a969-c8c851af345d-kube-api-access-bl4c2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:01.192705 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:01.192676    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"
Oct 30 20:55:01.193202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:01.192724    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf" event={"ID":"3d3a2612-d065-4443-a969-c8c851af345d","Type":"ContainerDied","Data":"080cfedf172e074298520b06146e1344dff07ef57be54524427ffbec6ea6a50f"}
Oct 30 20:55:01.193202 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:01.192793    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080cfedf172e074298520b06146e1344dff07ef57be54524427ffbec6ea6a50f"
Oct 30 20:55:01.543947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:01.543921    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" path="/var/lib/kubelet/pods/45f81c78-fe7a-4e77-afbb-fd21816db221/volumes"
Oct 30 20:55:01.981120 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:01.981090    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:55:07.434055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:07.434019    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"]
Oct 30 20:55:07.482716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:07.482685    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab9c38-mbv27"]
Oct 30 20:55:07.543749 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:07.543718    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405b067f-65ff-4a61-8c33-80dbd82eb590" path="/var/lib/kubelet/pods/405b067f-65ff-4a61-8c33-80dbd82eb590/volumes"
Oct 30 20:55:13.792238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:13.792207    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"]
Oct 30 20:55:13.840708 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:13.840683    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8ac60-54thr"]
Oct 30 20:55:14.755152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:14.755118    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"]
Oct 30 20:55:14.755404 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:14.755348    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy" containerID="cri-o://4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc" gracePeriod=30
Oct 30 20:55:15.045852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:15.045767    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"]
Oct 30 20:55:15.048707 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:15.048680    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389ffc2a-xnpt9"]
Oct 30 20:55:15.543671 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:15.543639    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3772bdaa-a3ac-441c-b46b-2dc0573937de" path="/var/lib/kubelet/pods/3772bdaa-a3ac-441c-b46b-2dc0573937de/volumes"
Oct 30 20:55:15.544148 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:15.544133    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8877f4ed-d319-42f6-976e-928c229f6008" path="/var/lib/kubelet/pods/8877f4ed-d319-42f6-976e-928c229f6008/volumes"
Oct 30 20:55:20.053605 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.053580    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:55:20.171009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.170979    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171024    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171064    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171084    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171121    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171189    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171222    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171387 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171260    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171548 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171363    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:20.171548 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171421    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs\") pod \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\" (UID: \"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031\") "
Oct 30 20:55:20.171548 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171446    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:20.171728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171586    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:20.171728 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171717    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:20.171930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.171907    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:20.172949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.172926    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.173019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.172955    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.173019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.172968    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.173019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.172981    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.173019 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.172996    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.173487 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.173431    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "kube-api-access-hsn67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:20.173721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.173700    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:20.174044 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.174001    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:55:20.174044 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.174023    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token" pod "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" (UID: "ad9bc53a-f13d-4a2e-9aeb-bd6debf23031"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:20.274003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.273931    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsn67\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-kube-api-access-hsn67\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.274003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.273958    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.274003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.273968    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.274003 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.273976    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:20.383371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.383338    5967 generic.go:356] "Generic (PLEG): container finished" podID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerID="4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc" exitCode=0
Oct 30 20:55:20.383540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.383388    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"
Oct 30 20:55:20.383540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.383429    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" event={"ID":"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031","Type":"ContainerDied","Data":"4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc"}
Oct 30 20:55:20.383540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.383487    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf" event={"ID":"ad9bc53a-f13d-4a2e-9aeb-bd6debf23031","Type":"ContainerDied","Data":"c65f5ad2adafb5b94d1b6f117307db6781e8b77ab38dcd659d3e5a48e4406e79"}
Oct 30 20:55:20.383540 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.383515    5967 scope.go:122] "RemoveContainer" containerID="4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc"
Oct 30 20:55:20.397174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.397160    5967 scope.go:122] "RemoveContainer" containerID="a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"
Oct 30 20:55:20.408460 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.408424    5967 scope.go:122] "RemoveContainer" containerID="4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc"
Oct 30 20:55:20.408826 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:20.408803    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc\": container with ID starting with 4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc not found: ID does not exist" containerID="4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc"
Oct 30 20:55:20.408914 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.408835    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc"} err="failed to get container status \"4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc\": rpc error: code = NotFound desc = could not find container \"4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc\": container with ID starting with 4f67621dcef59f793e87457b88244e7e571787f3da59ff0ec0be96eded441bcc not found: ID does not exist"
Oct 30 20:55:20.408914 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.408858    5967 scope.go:122] "RemoveContainer" containerID="a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"
Oct 30 20:55:20.409273 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:20.409249    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575\": container with ID starting with a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575 not found: ID does not exist" containerID="a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"
Oct 30 20:55:20.409365 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.409279    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575"} err="failed to get container status \"a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575\": rpc error: code = NotFound desc = could not find container \"a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575\": container with ID starting with a77204d2578ce3ffab0f79bd1f08dc287fcdbb80ec634902ef14e0b3349fc575 not found: ID does not exist"
Oct 30 20:55:20.552096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.552025    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"]
Oct 30 20:55:20.557464 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.557439    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf"]
Oct 30 20:55:20.763217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.763186    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"]
Oct 30 20:55:20.763388 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:20.763365    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" containerID="cri-o://3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0" gracePeriod=30
Oct 30 20:55:21.544057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:21.544020    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" path="/var/lib/kubelet/pods/ad9bc53a-f13d-4a2e-9aeb-bd6debf23031/volumes"
Oct 30 20:55:21.558564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:21.558541    5967 scope.go:122] "RemoveContainer" containerID="bb95572cab973b3907738478db6c35b631f7af9779a29a27d3f5c0f4ec134222"
Oct 30 20:55:21.568826 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:21.568807    5967 scope.go:122] "RemoveContainer" containerID="619387df1f960bf34383665414f1a33e29656537b08fd38c018e864d34cbee91"
Oct 30 20:55:21.903441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:21.903357    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:55:21.903441 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:21.903419    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:55:26.038862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.038793    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:55:26.160753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.160705    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.160911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.160783    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.160911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.160817    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.160911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.160853    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.160911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.160909    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.161164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.160966    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.161164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161055    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.161164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161105    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.161164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161138    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:26.161164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161158    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert\") pod \"b6926a1c-820c-48b6-b3fd-31b4026a7804\" (UID: \"b6926a1c-820c-48b6-b3fd-31b4026a7804\") "
Oct 30 20:55:26.161423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161282    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:26.161423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161372    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:26.161423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161398    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:26.161693 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161673    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:26.161832 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161814    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.161905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161838    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.161905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161851    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.161905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161865    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.161905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.161878    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6926a1c-820c-48b6-b3fd-31b4026a7804-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.163115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.163094    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:55:26.163396 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.163377    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "kube-api-access-njfcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:26.163592 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.163568    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:26.163685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.163627    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token" pod "b6926a1c-820c-48b6-b3fd-31b4026a7804" (UID: "b6926a1c-820c-48b6-b3fd-31b4026a7804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:26.262961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.262929    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.262961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.262955    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.262961 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.262964    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.263178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.262972    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njfcq\" (UniqueName: \"kubernetes.io/projected/b6926a1c-820c-48b6-b3fd-31b4026a7804-kube-api-access-njfcq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:26.460814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.455717    5967 generic.go:356] "Generic (PLEG): container finished" podID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerID="3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0" exitCode=0
Oct 30 20:55:26.460814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.455858    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" event={"ID":"b6926a1c-820c-48b6-b3fd-31b4026a7804","Type":"ContainerDied","Data":"3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0"}
Oct 30 20:55:26.460814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.455911    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h" event={"ID":"b6926a1c-820c-48b6-b3fd-31b4026a7804","Type":"ContainerDied","Data":"e02d1ec4f21434677e26ae4f9317a489895a33ec69582c966e861531debec223"}
Oct 30 20:55:26.460814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.455942    5967 scope.go:122] "RemoveContainer" containerID="3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0"
Oct 30 20:55:26.460814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.456096    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"
Oct 30 20:55:26.482568 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.482266    5967 scope.go:122] "RemoveContainer" containerID="6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"
Oct 30 20:55:26.497711 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.497612    5967 scope.go:122] "RemoveContainer" containerID="3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0"
Oct 30 20:55:26.498018 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:26.497992    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0\": container with ID starting with 3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0 not found: ID does not exist" containerID="3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0"
Oct 30 20:55:26.498243 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.498031    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0"} err="failed to get container status \"3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0\": rpc error: code = NotFound desc = could not find container \"3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0\": container with ID starting with 3e707500c67b084fa57065eb2acabbaef4c8aa8bac97f7989ed1cf9db8744fd0 not found: ID does not exist"
Oct 30 20:55:26.498243 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.498054    5967 scope.go:122] "RemoveContainer" containerID="6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"
Oct 30 20:55:26.498385 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:26.498329    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075\": container with ID starting with 6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075 not found: ID does not exist" containerID="6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"
Oct 30 20:55:26.498385 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.498358    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075"} err="failed to get container status \"6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075\": rpc error: code = NotFound desc = could not find container \"6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075\": container with ID starting with 6fb113045f5d5bc7e5495294adf1f453ecbc008bab56cd44f41497ac66a9c075 not found: ID does not exist"
Oct 30 20:55:26.628627 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.628591    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"]
Oct 30 20:55:26.634599 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:26.634569    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h"]
Oct 30 20:55:27.542484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:27.542456    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" path="/var/lib/kubelet/pods/b6926a1c-820c-48b6-b3fd-31b4026a7804/volumes"
Oct 30 20:55:30.476849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:30.476802    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"]
Oct 30 20:55:30.503444 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:30.503408    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d71c23-nczvf"]
Oct 30 20:55:31.543776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:31.543724    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3a2612-d065-4443-a969-c8c851af345d" path="/var/lib/kubelet/pods/3d3a2612-d065-4443-a969-c8c851af345d/volumes"
Oct 30 20:55:31.980857 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:31.980824    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:55:37.109608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.109575    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"]
Oct 30 20:55:37.110121 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.109857    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" containerID="cri-o://1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6" gracePeriod=30
Oct 30 20:55:37.958530 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.958498    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"]
Oct 30 20:55:37.959186 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959167    5967 cpu_manager.go:414] "containerMap: removing container" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy"
Oct 30 20:55:37.959186 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959187    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959204    5967 cpu_manager.go:414] "containerMap: removing container" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959211    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959290    5967 cpu_manager.go:414] "containerMap: removing container" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959298    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959315    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959322    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959357    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy"
Oct 30 20:55:37.959371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959364    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959376    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3d3a2612-d065-4443-a969-c8c851af345d" containerName="revoke-keys"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959383    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3a2612-d065-4443-a969-c8c851af345d" containerName="revoke-keys"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959403    5967 cpu_manager.go:414] "containerMap: removing container" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959410    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959572    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959585    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3a2612-d065-4443-a969-c8c851af345d" containerName="revoke-keys"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959596    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959612    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959622    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6926a1c-820c-48b6-b3fd-31b4026a7804" containerName="istio-proxy"
Oct 30 20:55:37.959814 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.959637    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f81c78-fe7a-4e77-afbb-fd21816db221" containerName="istio-proxy"
Oct 30 20:55:37.964770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:37.964718    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.020872 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.020845    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"]
Oct 30 20:55:38.065721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.065694    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:38.065721 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.065709    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:55:38.065900 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.065755    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:55:38.065900 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.065793    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:55:38.069481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.069458    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs8xz\" (UniqueName: \"kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.069624 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.069605    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.069696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.069683    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.171218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.171186    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs8xz\" (UniqueName: \"kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.171623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.171318    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.171623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.171385    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.171981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.171953    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.173795 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.173766    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.227310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.227289    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs8xz\" (UniqueName: \"kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.277973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.277952    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:38.362926 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.362889    5967 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:55:38.363063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.362955    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:55:38.654250 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:55:38.654210    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4648092d_e5ee_4c0c_a136_1a349a4d2a66.slice/crio-6a8befb8a2822773b57c80c9c49ec3383c1f878c4a7ac42227534efbcbf079b9 WatchSource:0}: Error finding container 6a8befb8a2822773b57c80c9c49ec3383c1f878c4a7ac42227534efbcbf079b9: Status 404 returned error can't find the container with id 6a8befb8a2822773b57c80c9c49ec3383c1f878c4a7ac42227534efbcbf079b9
Oct 30 20:55:38.654394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:38.654371    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"]
Oct 30 20:55:39.565903 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:39.565877    5967 generic.go:356] "Generic (PLEG): container finished" podID="4648092d-e5ee-4c0c-a136-1a349a4d2a66" containerID="6dfe993e411a188926edaa523ab064580336b2a02186b362df16595797ed9e9d" exitCode=0
Oct 30 20:55:39.566188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:39.565958    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w" event={"ID":"4648092d-e5ee-4c0c-a136-1a349a4d2a66","Type":"ContainerDied","Data":"6dfe993e411a188926edaa523ab064580336b2a02186b362df16595797ed9e9d"}
Oct 30 20:55:39.566188 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:39.565989    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w" event={"ID":"4648092d-e5ee-4c0c-a136-1a349a4d2a66","Type":"ContainerStarted","Data":"6a8befb8a2822773b57c80c9c49ec3383c1f878c4a7ac42227534efbcbf079b9"}
Oct 30 20:55:40.730192 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.730165    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:40.800928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.800901    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz\" (UniqueName: \"kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz\") pod \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") "
Oct 30 20:55:40.801049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.800939    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca\") pod \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") "
Oct 30 20:55:40.801049 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.800987    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca\") pod \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\" (UID: \"4648092d-e5ee-4c0c-a136-1a349a4d2a66\") "
Oct 30 20:55:40.801391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.801369    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca" pod "4648092d-e5ee-4c0c-a136-1a349a4d2a66" (UID: "4648092d-e5ee-4c0c-a136-1a349a4d2a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:40.801642 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.801594    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:40.803028 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.803009    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz" pod "4648092d-e5ee-4c0c-a136-1a349a4d2a66" (UID: "4648092d-e5ee-4c0c-a136-1a349a4d2a66"). InnerVolumeSpecName "kube-api-access-vs8xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:40.803152 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.803136    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca" pod "4648092d-e5ee-4c0c-a136-1a349a4d2a66" (UID: "4648092d-e5ee-4c0c-a136-1a349a4d2a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:40.903319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.903241    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs8xz\" (UniqueName: \"kubernetes.io/projected/4648092d-e5ee-4c0c-a136-1a349a4d2a66-kube-api-access-vs8xz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:40.903319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:40.903268    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4648092d-e5ee-4c0c-a136-1a349a4d2a66-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:41.591676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:41.591650    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"
Oct 30 20:55:41.591845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:41.591696    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w" event={"ID":"4648092d-e5ee-4c0c-a136-1a349a4d2a66","Type":"ContainerDied","Data":"6a8befb8a2822773b57c80c9c49ec3383c1f878c4a7ac42227534efbcbf079b9"}
Oct 30 20:55:41.591845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:41.591777    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8befb8a2822773b57c80c9c49ec3383c1f878c4a7ac42227534efbcbf079b9"
Oct 30 20:55:42.378349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.378297    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:55:42.421335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421309    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421415    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421438    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421457    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421525 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421482    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421523    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421606 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421568    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:42.421676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421603    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421657    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421735    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:42.421819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421787    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data\") pod \"994aa1e0-293c-4c27-9d9f-b402607af934\" (UID: \"994aa1e0-293c-4c27-9d9f-b402607af934\") "
Oct 30 20:55:42.421928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421868    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:42.421983 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.421924    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:42.422053 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.422024    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:42.422555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.422533    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.422685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.422560    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.422685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.422575    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.422685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.422590    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994aa1e0-293c-4c27-9d9f-b402607af934-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.422685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.422604    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.423773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.423720    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "kube-api-access-zkf5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:42.423877 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.423828    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:55:42.423929 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.423895    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:42.424042 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.424021    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token" pod "994aa1e0-293c-4c27-9d9f-b402607af934" (UID: "994aa1e0-293c-4c27-9d9f-b402607af934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:42.524410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.524381    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994aa1e0-293c-4c27-9d9f-b402607af934-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.524410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.524405    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994aa1e0-293c-4c27-9d9f-b402607af934-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.524410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.524414    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.524604 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.524424    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkf5d\" (UniqueName: \"kubernetes.io/projected/994aa1e0-293c-4c27-9d9f-b402607af934-kube-api-access-zkf5d\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.601262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.601233    5967 generic.go:356] "Generic (PLEG): container finished" podID="994aa1e0-293c-4c27-9d9f-b402607af934" containerID="1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6" exitCode=0
Oct 30 20:55:42.601431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.601284    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"
Oct 30 20:55:42.601431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.601314    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" event={"ID":"994aa1e0-293c-4c27-9d9f-b402607af934","Type":"ContainerDied","Data":"1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6"}
Oct 30 20:55:42.601431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.601369    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p" event={"ID":"994aa1e0-293c-4c27-9d9f-b402607af934","Type":"ContainerDied","Data":"ad258ba51d65cb121754b59097b057775ab77d0272e40df64c3deca1cb961ed9"}
Oct 30 20:55:42.601431 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.601398    5967 scope.go:122] "RemoveContainer" containerID="1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6"
Oct 30 20:55:42.613024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.613008    5967 scope.go:122] "RemoveContainer" containerID="7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"
Oct 30 20:55:42.622372 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.622356    5967 scope.go:122] "RemoveContainer" containerID="1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6"
Oct 30 20:55:42.622640 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:42.622624    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6\": container with ID starting with 1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6 not found: ID does not exist" containerID="1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6"
Oct 30 20:55:42.622716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.622652    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6"} err="failed to get container status \"1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6\": rpc error: code = NotFound desc = could not find container \"1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6\": container with ID starting with 1cdf7c1b7dff6fc13aa783eddf1d0e2c84048eef666e5c630d6dae7a29acf2f6 not found: ID does not exist"
Oct 30 20:55:42.622716 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.622676    5967 scope.go:122] "RemoveContainer" containerID="7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"
Oct 30 20:55:42.622954 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:55:42.622936    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf\": container with ID starting with 7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf not found: ID does not exist" containerID="7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"
Oct 30 20:55:42.622994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.622961    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf"} err="failed to get container status \"7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf\": rpc error: code = NotFound desc = could not find container \"7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf\": container with ID starting with 7d3dd49aac4ce2a9e22e914aa12862791b735173cb4727cd9db4804a9f6cdacf not found: ID does not exist"
Oct 30 20:55:42.765932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.765900    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"]
Oct 30 20:55:42.771127 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:42.771102    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p"]
Oct 30 20:55:43.543608 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:55:43.543575    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" path="/var/lib/kubelet/pods/994aa1e0-293c-4c27-9d9f-b402607af934/volumes"
Oct 30 20:56:01.981145 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:56:01.981109    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:56:07.155348 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:07.155316    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"]
Oct 30 20:56:07.203784 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:07.203755    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712099-gctb6"]
Oct 30 20:56:07.543301 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:07.543270    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4b1aa9-df57-47b7-8511-ecc36ff851ba" path="/var/lib/kubelet/pods/2b4b1aa9-df57-47b7-8511-ecc36ff851ba/volumes"
Oct 30 20:56:13.998093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:13.998052    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"]
Oct 30 20:56:13.998577 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:13.998309    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" containerID="cri-o://6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4" gracePeriod=30
Oct 30 20:56:16.448201 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:16.448161    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:56:16.448596 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:16.448222    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:56:19.271096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.271037    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:56:19.336207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336181    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336225    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336278    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336313    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336375    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336454 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336421    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:19.336551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336481    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336542    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336643 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336620    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336667    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy\") pod \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\" (UID: \"25a34841-d0e2-40dd-b4cc-8343e42b6da2\") "
Oct 30 20:56:19.336702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336678    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:19.336869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336807    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:19.336979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336934    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:19.336979 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.336964    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:19.337338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.337320    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.337338 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.337340    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.337719 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.337350    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.337719 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.337359    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.337719 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.337368    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.339016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.338992    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:19.339205 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.339185    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:19.339286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.339270    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "kube-api-access-4g6sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:19.339324 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.339270    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo" pod "25a34841-d0e2-40dd-b4cc-8343e42b6da2" (UID: "25a34841-d0e2-40dd-b4cc-8343e42b6da2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:56:19.438950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.438915    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.438950 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.438951    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g6sn\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-kube-api-access-4g6sn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.439218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.438966    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.439218 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.438980    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a34841-d0e2-40dd-b4cc-8343e42b6da2-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:19.932868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.932799    5967 generic.go:356] "Generic (PLEG): container finished" podID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerID="6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4" exitCode=0
Oct 30 20:56:19.932868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.932849    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"
Oct 30 20:56:19.933052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.932880    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" event={"ID":"25a34841-d0e2-40dd-b4cc-8343e42b6da2","Type":"ContainerDied","Data":"6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4"}
Oct 30 20:56:19.933052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.932934    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml" event={"ID":"25a34841-d0e2-40dd-b4cc-8343e42b6da2","Type":"ContainerDied","Data":"4fce32b064f93dca19589cb56b5c3109e68ebf6fee47bf4662a4c278d31704ff"}
Oct 30 20:56:19.933052 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.932957    5967 scope.go:122] "RemoveContainer" containerID="6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4"
Oct 30 20:56:19.943349 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.943334    5967 scope.go:122] "RemoveContainer" containerID="7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"
Oct 30 20:56:19.953109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.953085    5967 scope.go:122] "RemoveContainer" containerID="6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4"
Oct 30 20:56:19.953395 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:56:19.953367    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4\": container with ID starting with 6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4 not found: ID does not exist" containerID="6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4"
Oct 30 20:56:19.953445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.953404    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4"} err="failed to get container status \"6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4\": rpc error: code = NotFound desc = could not find container \"6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4\": container with ID starting with 6c3bdc51d7dc6c52803081a22d2f9f82ebdcacc0f36a2848a6547b47cb1295e4 not found: ID does not exist"
Oct 30 20:56:19.953445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.953423    5967 scope.go:122] "RemoveContainer" containerID="7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"
Oct 30 20:56:19.953696 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:56:19.953678    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725\": container with ID starting with 7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725 not found: ID does not exist" containerID="7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"
Oct 30 20:56:19.953775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:19.953704    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725"} err="failed to get container status \"7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725\": rpc error: code = NotFound desc = could not find container \"7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725\": container with ID starting with 7db4bb5330fe38b4ff507e14a2b3fe2effc4529c59066d9b95e4ea6ac7f19725 not found: ID does not exist"
Oct 30 20:56:20.099971 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:20.099943    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"]
Oct 30 20:56:20.113418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:20.113389    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml"]
Oct 30 20:56:21.543285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:21.543252    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" path="/var/lib/kubelet/pods/25a34841-d0e2-40dd-b4cc-8343e42b6da2/volumes"
Oct 30 20:56:21.620203 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:21.620179    5967 scope.go:122] "RemoveContainer" containerID="fed4a782f13e9be23dcdd4f32a657e6c814e85ae194e1d852c417f88a91baae0"
Oct 30 20:56:31.980543 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:56:31.980460    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:56:33.525938 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.525906    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"]
Oct 30 20:56:33.526481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526467    5967 cpu_manager.go:414] "containerMap: removing container" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy"
Oct 30 20:56:33.526531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526482    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy"
Oct 30 20:56:33.526531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526501    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4648092d-e5ee-4c0c-a136-1a349a4d2a66" containerName="revoke-keys"
Oct 30 20:56:33.526531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526505    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4648092d-e5ee-4c0c-a136-1a349a4d2a66" containerName="revoke-keys"
Oct 30 20:56:33.526531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526513    5967 cpu_manager.go:414] "containerMap: removing container" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy"
Oct 30 20:56:33.526531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526518    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526536    5967 cpu_manager.go:414] "containerMap: removing container" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526541    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526560    5967 cpu_manager.go:414] "containerMap: removing container" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526566    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526656    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526665    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4648092d-e5ee-4c0c-a136-1a349a4d2a66" containerName="revoke-keys"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526673    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526680    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9bc53a-f13d-4a2e-9aeb-bd6debf23031" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526686    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a34841-d0e2-40dd-b4cc-8343e42b6da2" containerName="istio-proxy"
Oct 30 20:56:33.526700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.526694    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="994aa1e0-293c-4c27-9d9f-b402607af934" containerName="istio-proxy"
Oct 30 20:56:33.531658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.531643    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.587644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.587620    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"]
Oct 30 20:56:33.633270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.633240    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:56:33.633270 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.633253    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:56:33.633492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.633299    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:56:33.633492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.633455    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:56:33.650057 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.650022    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.650195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.650175    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.650293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.650280    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjfx\" (UniqueName: \"kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.751189 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.751160    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.751316 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.751244    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjfx\" (UniqueName: \"kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.751376 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.751312    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.751908 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.751881    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.753589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.753569    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.807817 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.807729    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjfx\" (UniqueName: \"kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx\") pod \"maas-api-revoke-keys-e2e-mt-27219c73-nrv5r\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:33.844850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:33.844825    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:34.223423 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:34.223397    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"]
Oct 30 20:56:34.224869 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:56:34.224846    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3db7ff_db4d_4840_a24b_b02b26ae182b.slice/crio-c35b0fcc84a3e8cb407906335121d956c20ee737a102c3e69b0b3216d5d1a99e WatchSource:0}: Error finding container c35b0fcc84a3e8cb407906335121d956c20ee737a102c3e69b0b3216d5d1a99e: Status 404 returned error can't find the container with id c35b0fcc84a3e8cb407906335121d956c20ee737a102c3e69b0b3216d5d1a99e
Oct 30 20:56:35.042354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:35.042325    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r" event={"ID":"9d3db7ff-db4d-4840-a24b-b02b26ae182b","Type":"ContainerStarted","Data":"e4fa8b1010736509cbd70a7bc8e9108a818f48ad85d2e9a8c1d5727e5a009c96"}
Oct 30 20:56:35.042731 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:35.042375    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r" event={"ID":"9d3db7ff-db4d-4840-a24b-b02b26ae182b","Type":"ContainerStarted","Data":"c35b0fcc84a3e8cb407906335121d956c20ee737a102c3e69b0b3216d5d1a99e"}
Oct 30 20:56:35.157626 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:35.157576    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r" podStartSLOduration=2.15756242 podStartE2EDuration="2.15756242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:35.155126013 +0000 UTC m=+2175.755927974" watchObservedRunningTime="2026-10-30 20:56:35.15756242 +0000 UTC m=+2175.758364376"
Oct 30 20:56:36.051222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:36.051185    5967 generic.go:356] "Generic (PLEG): container finished" podID="9d3db7ff-db4d-4840-a24b-b02b26ae182b" containerID="e4fa8b1010736509cbd70a7bc8e9108a818f48ad85d2e9a8c1d5727e5a009c96" exitCode=0
Oct 30 20:56:36.051631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:36.051263    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r" event={"ID":"9d3db7ff-db4d-4840-a24b-b02b26ae182b","Type":"ContainerDied","Data":"e4fa8b1010736509cbd70a7bc8e9108a818f48ad85d2e9a8c1d5727e5a009c96"}
Oct 30 20:56:37.241183 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.241160    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:37.299212 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.299186    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca\") pod \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") "
Oct 30 20:56:37.299354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.299271    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx\" (UniqueName: \"kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx\") pod \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") "
Oct 30 20:56:37.299408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.299370    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca\") pod \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\" (UID: \"9d3db7ff-db4d-4840-a24b-b02b26ae182b\") "
Oct 30 20:56:37.299695 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.299671    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca" pod "9d3db7ff-db4d-4840-a24b-b02b26ae182b" (UID: "9d3db7ff-db4d-4840-a24b-b02b26ae182b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:37.299991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.299875    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:37.301375 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.301351    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx" pod "9d3db7ff-db4d-4840-a24b-b02b26ae182b" (UID: "9d3db7ff-db4d-4840-a24b-b02b26ae182b"). InnerVolumeSpecName "kube-api-access-2kjfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:37.301471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.301380    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca" pod "9d3db7ff-db4d-4840-a24b-b02b26ae182b" (UID: "9d3db7ff-db4d-4840-a24b-b02b26ae182b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:37.401507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.401472    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3db7ff-db4d-4840-a24b-b02b26ae182b-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:37.401507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:37.401511    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kjfx\" (UniqueName: \"kubernetes.io/projected/9d3db7ff-db4d-4840-a24b-b02b26ae182b-kube-api-access-2kjfx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:38.066520 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:38.066490    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"
Oct 30 20:56:38.066709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:38.066526    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r" event={"ID":"9d3db7ff-db4d-4840-a24b-b02b26ae182b","Type":"ContainerDied","Data":"c35b0fcc84a3e8cb407906335121d956c20ee737a102c3e69b0b3216d5d1a99e"}
Oct 30 20:56:38.066709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:38.066570    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c35b0fcc84a3e8cb407906335121d956c20ee737a102c3e69b0b3216d5d1a99e"
Oct 30 20:56:53.358302 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:53.358259    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"]
Oct 30 20:56:53.407136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:53.407110    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-061bda-m5j8w"]
Oct 30 20:56:53.544869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:53.544842    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4648092d-e5ee-4c0c-a136-1a349a4d2a66" path="/var/lib/kubelet/pods/4648092d-e5ee-4c0c-a136-1a349a4d2a66/volumes"
Oct 30 20:56:59.846805 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:59.846772    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"]
Oct 30 20:56:59.847210 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:56:59.846973    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" containerID="cri-o://7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011" gracePeriod=30
Oct 30 20:57:00.811347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.811316    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"]
Oct 30 20:57:00.811972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.811957    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9d3db7ff-db4d-4840-a24b-b02b26ae182b" containerName="revoke-keys"
Oct 30 20:57:00.812025 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.811974    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3db7ff-db4d-4840-a24b-b02b26ae182b" containerName="revoke-keys"
Oct 30 20:57:00.812105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.812096    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3db7ff-db4d-4840-a24b-b02b26ae182b" containerName="revoke-keys"
Oct 30 20:57:00.815928 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.815903    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:00.916366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.916338    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nsjj\" (UniqueName: \"kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:00.916753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.916410    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:00.916753 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.916475    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:00.917342 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.917321    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:00.917474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.917415    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:57:00.917474 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.917443    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:57:00.917564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.917482    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:57:00.920285 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:00.920263    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"]
Oct 30 20:57:01.017286 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.017262    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.017433 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.017302    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.017513 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.017438    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nsjj\" (UniqueName: \"kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.018020 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.017997    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.019798 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.019716    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.074428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.074372    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nsjj\" (UniqueName: \"kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.131027 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.131009    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:01.505008 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:01.504978    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"]
Oct 30 20:57:01.506271 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:57:01.506246    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0738015_c327_417a_ba2f_a371dd5ff212.slice/crio-5556c92314bc672fea46abc0feb4b7d51a832eadb5a03eaeddb2fc0f2133fe67 WatchSource:0}: Error finding container 5556c92314bc672fea46abc0feb4b7d51a832eadb5a03eaeddb2fc0f2133fe67: Status 404 returned error can't find the container with id 5556c92314bc672fea46abc0feb4b7d51a832eadb5a03eaeddb2fc0f2133fe67
Oct 30 20:57:01.980703 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:01.980670    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:57:02.270277 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:02.270248    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k" event={"ID":"e0738015-c327-417a-ba2f-a371dd5ff212","Type":"ContainerStarted","Data":"70cd0b81e624e080e2816bfc516259b8c9b890e0a17dce5ab4819eb0b5f8e019"}
Oct 30 20:57:02.270461 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:02.270297    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k" event={"ID":"e0738015-c327-417a-ba2f-a371dd5ff212","Type":"ContainerStarted","Data":"5556c92314bc672fea46abc0feb4b7d51a832eadb5a03eaeddb2fc0f2133fe67"}
Oct 30 20:57:02.387877 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:02.387833    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k" podStartSLOduration=2.387819593 podStartE2EDuration="2.387819593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:02.385327167 +0000 UTC m=+2202.986129127" watchObservedRunningTime="2026-10-30 20:57:02.387819593 +0000 UTC m=+2202.988621587"
Oct 30 20:57:02.908426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:02.908390    5967 patch_prober.go:29] interesting pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:57:02.908597 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:02.908457    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:57:03.279119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:03.279084    5967 generic.go:356] "Generic (PLEG): container finished" podID="e0738015-c327-417a-ba2f-a371dd5ff212" containerID="70cd0b81e624e080e2816bfc516259b8c9b890e0a17dce5ab4819eb0b5f8e019" exitCode=0
Oct 30 20:57:03.279527 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:03.279182    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k" event={"ID":"e0738015-c327-417a-ba2f-a371dd5ff212","Type":"ContainerDied","Data":"70cd0b81e624e080e2816bfc516259b8c9b890e0a17dce5ab4819eb0b5f8e019"}
Oct 30 20:57:04.439505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.439478    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:04.572227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.572149    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca\") pod \"e0738015-c327-417a-ba2f-a371dd5ff212\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") "
Oct 30 20:57:04.572227 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.572212    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj\" (UniqueName: \"kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj\") pod \"e0738015-c327-417a-ba2f-a371dd5ff212\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") "
Oct 30 20:57:04.572445 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.572269    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca\") pod \"e0738015-c327-417a-ba2f-a371dd5ff212\" (UID: \"e0738015-c327-417a-ba2f-a371dd5ff212\") "
Oct 30 20:57:04.572612 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.572581    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca" pod "e0738015-c327-417a-ba2f-a371dd5ff212" (UID: "e0738015-c327-417a-ba2f-a371dd5ff212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:04.572864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.572842    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:04.574468 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.574441    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca" pod "e0738015-c327-417a-ba2f-a371dd5ff212" (UID: "e0738015-c327-417a-ba2f-a371dd5ff212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:04.574579 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.574564    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj" pod "e0738015-c327-417a-ba2f-a371dd5ff212" (UID: "e0738015-c327-417a-ba2f-a371dd5ff212"). InnerVolumeSpecName "kube-api-access-7nsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:04.674585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.674548    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nsjj\" (UniqueName: \"kubernetes.io/projected/e0738015-c327-417a-ba2f-a371dd5ff212-kube-api-access-7nsjj\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:04.674585 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:04.674582    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0738015-c327-417a-ba2f-a371dd5ff212-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.105266 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.105240    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:57:05.180115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180050    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180089    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180115 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180112    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180184    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180205    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180227    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180257    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180382    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180497    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs\") pod \"d2343b12-8707-454e-bdce-61f7ba30e16e\" (UID: \"d2343b12-8707-454e-bdce-61f7ba30e16e\") "
Oct 30 20:57:05.180670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180509    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:05.180670 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180527    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:05.180876 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180788    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:05.180876 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.180825    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:05.181263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.181199    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.181263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.181223    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343b12-8707-454e-bdce-61f7ba30e16e-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.181263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.181233    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.181263 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.181242    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.181517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.181495    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:05.182619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.182594    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "kube-api-access-vgvbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:05.182992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.182964    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:05.183098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.183039    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:05.183098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.183088    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo" pod "d2343b12-8707-454e-bdce-61f7ba30e16e" (UID: "d2343b12-8707-454e-bdce-61f7ba30e16e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:05.283063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.283037    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.283063 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.283061    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.283207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.283070    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.283207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.283080    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343b12-8707-454e-bdce-61f7ba30e16e-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.283207 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.283088    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgvbb\" (UniqueName: \"kubernetes.io/projected/d2343b12-8707-454e-bdce-61f7ba30e16e-kube-api-access-vgvbb\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:05.294909 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.294882    5967 generic.go:356] "Generic (PLEG): container finished" podID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerID="7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011" exitCode=0
Oct 30 20:57:05.295021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.294924    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" event={"ID":"d2343b12-8707-454e-bdce-61f7ba30e16e","Type":"ContainerDied","Data":"7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011"}
Oct 30 20:57:05.295021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.294975    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t" event={"ID":"d2343b12-8707-454e-bdce-61f7ba30e16e","Type":"ContainerDied","Data":"a080fb7e4563e7d9ec70cb600ded1e5f9db4af88908e0e020e09edab18ad8afb"}
Oct 30 20:57:05.295021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.295001    5967 scope.go:122] "RemoveContainer" containerID="7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011"
Oct 30 20:57:05.295021 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.294931    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"
Oct 30 20:57:05.296678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.296645    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"
Oct 30 20:57:05.296865 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.296698    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k" event={"ID":"e0738015-c327-417a-ba2f-a371dd5ff212","Type":"ContainerDied","Data":"5556c92314bc672fea46abc0feb4b7d51a832eadb5a03eaeddb2fc0f2133fe67"}
Oct 30 20:57:05.296865 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.296771    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5556c92314bc672fea46abc0feb4b7d51a832eadb5a03eaeddb2fc0f2133fe67"
Oct 30 20:57:05.306924 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.306909    5967 scope.go:122] "RemoveContainer" containerID="8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"
Oct 30 20:57:05.317010 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.316995    5967 scope.go:122] "RemoveContainer" containerID="7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011"
Oct 30 20:57:05.317268 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:05.317247    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011\": container with ID starting with 7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011 not found: ID does not exist" containerID="7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011"
Oct 30 20:57:05.317327 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.317274    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011"} err="failed to get container status \"7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011\": rpc error: code = NotFound desc = could not find container \"7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011\": container with ID starting with 7477a2dc3d9b59ab98669bcf113bbc1b726450395b27be05fe6eb02dcab8f011 not found: ID does not exist"
Oct 30 20:57:05.317327 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.317292    5967 scope.go:122] "RemoveContainer" containerID="8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"
Oct 30 20:57:05.317576 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:05.317554    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d\": container with ID starting with 8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d not found: ID does not exist" containerID="8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"
Oct 30 20:57:05.317621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.317586    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d"} err="failed to get container status \"8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d\": rpc error: code = NotFound desc = could not find container \"8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d\": container with ID starting with 8cd9b73814979c69f3ec68c68591cf831a91d53fa129482dd51976edacf0990d not found: ID does not exist"
Oct 30 20:57:05.346555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.346529    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"]
Oct 30 20:57:05.542117 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.542082    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"]
Oct 30 20:57:05.551769 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:05.551730    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t"]
Oct 30 20:57:07.546887 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:07.546849    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" path="/var/lib/kubelet/pods/d2343b12-8707-454e-bdce-61f7ba30e16e/volumes"
Oct 30 20:57:12.334508 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:12.334477    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"]
Oct 30 20:57:12.442624 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:12.442580    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"]
Oct 30 20:57:17.335574 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:17.335519    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerName="payload-processing" containerID="cri-o://9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f" gracePeriod=25
Oct 30 20:57:17.443698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:17.443658    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerName="payload-pre-processing" containerID="cri-o://0435a2d9331ee04443a37260ef73594bd77db7246513fa880dcbe98ec3c8417a" gracePeriod=25
Oct 30 20:57:19.711146 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711117    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"]
Oct 30 20:57:19.711689 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711675    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy"
Oct 30 20:57:19.711734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711689    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy"
Oct 30 20:57:19.711734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711698    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e0738015-c327-417a-ba2f-a371dd5ff212" containerName="revoke-keys"
Oct 30 20:57:19.711734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711703    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0738015-c327-417a-ba2f-a371dd5ff212" containerName="revoke-keys"
Oct 30 20:57:19.711734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711718    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy"
Oct 30 20:57:19.711734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711722    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy"
Oct 30 20:57:19.711906 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711842    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy"
Oct 30 20:57:19.711906 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711851    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0738015-c327-417a-ba2f-a371dd5ff212" containerName="revoke-keys"
Oct 30 20:57:19.711906 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.711858    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2343b12-8707-454e-bdce-61f7ba30e16e" containerName="istio-proxy"
Oct 30 20:57:19.715269 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.715255    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.773564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.773537    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"]
Oct 30 20:57:19.802242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.802214    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xtw\" (UniqueName: \"kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.802411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.802290    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.802411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.802342    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.818912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.818886    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:57:19.818912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.818907    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:19.819111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.818955    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:57:19.819111 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.818999    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:57:19.903939 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.903913    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.904098 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.904082    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xtw\" (UniqueName: \"kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.904178 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.904155    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.904564 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.904543    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.906483 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.906459    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:19.961238 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:19.961188    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xtw\" (UniqueName: \"kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw\") pod \"maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:20.030457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.030431    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:20.368206 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.368163    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main" containerID="cri-o://7a7b4c6a7ff873103f80594034bf105f88560fbf6deb1c0b8dc4f6e60c91dbed" gracePeriod=45
Oct 30 20:57:20.409491 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.409467    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"]
Oct 30 20:57:20.427356 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:57:20.427330    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9d2402_6b56_49de_b2de_8f52f5ced372.slice/crio-5ff8b14ef05045671bacd28bd5766623c4280387540f7d9dc8b7cdea9b148c96 WatchSource:0}: Error finding container 5ff8b14ef05045671bacd28bd5766623c4280387540f7d9dc8b7cdea9b148c96: Status 404 returned error can't find the container with id 5ff8b14ef05045671bacd28bd5766623c4280387540f7d9dc8b7cdea9b148c96
Oct 30 20:57:20.487295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.487269    5967 generic.go:356] "Generic (PLEG): container finished" podID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerID="7a7b4c6a7ff873103f80594034bf105f88560fbf6deb1c0b8dc4f6e60c91dbed" exitCode=0
Oct 30 20:57:20.487426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.487342    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" event={"ID":"3c3b80af-9651-40cc-95aa-293d7d0ac560","Type":"ContainerDied","Data":"7a7b4c6a7ff873103f80594034bf105f88560fbf6deb1c0b8dc4f6e60c91dbed"}
Oct 30 20:57:20.489024 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.489004    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw" event={"ID":"1e9d2402-6b56-49de-b2de-8f52f5ced372","Type":"ContainerStarted","Data":"5ff8b14ef05045671bacd28bd5766623c4280387540f7d9dc8b7cdea9b148c96"}
Oct 30 20:57:20.620775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.620725    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:20.620775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.620760    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:20.628484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.628243    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:57:20.716138 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716104    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir\") pod \"3c3b80af-9651-40cc-95aa-293d7d0ac560\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") "
Oct 30 20:57:20.716531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716153    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8\" (UniqueName: \"kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8\") pod \"3c3b80af-9651-40cc-95aa-293d7d0ac560\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") "
Oct 30 20:57:20.716531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716186    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs\") pod \"3c3b80af-9651-40cc-95aa-293d7d0ac560\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") "
Oct 30 20:57:20.716531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716206    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm\") pod \"3c3b80af-9651-40cc-95aa-293d7d0ac560\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") "
Oct 30 20:57:20.716531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716233    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache\") pod \"3c3b80af-9651-40cc-95aa-293d7d0ac560\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") "
Oct 30 20:57:20.716531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716318    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home\") pod \"3c3b80af-9651-40cc-95aa-293d7d0ac560\" (UID: \"3c3b80af-9651-40cc-95aa-293d7d0ac560\") "
Oct 30 20:57:20.716531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716360    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir" pod "3c3b80af-9651-40cc-95aa-293d7d0ac560" (UID: "3c3b80af-9651-40cc-95aa-293d7d0ac560"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.716809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716533    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache" pod "3c3b80af-9651-40cc-95aa-293d7d0ac560" (UID: "3c3b80af-9651-40cc-95aa-293d7d0ac560"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.716809 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.716578    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home" pod "3c3b80af-9651-40cc-95aa-293d7d0ac560" (UID: "3c3b80af-9651-40cc-95aa-293d7d0ac560"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.717249 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.717228    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.717309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.717257    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.717309 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.717272    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.718699 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.718680    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm" pod "3c3b80af-9651-40cc-95aa-293d7d0ac560" (UID: "3c3b80af-9651-40cc-95aa-293d7d0ac560"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.718804 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.718713    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs" pod "3c3b80af-9651-40cc-95aa-293d7d0ac560" (UID: "3c3b80af-9651-40cc-95aa-293d7d0ac560"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:20.718804 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.718735    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8" pod "3c3b80af-9651-40cc-95aa-293d7d0ac560" (UID: "3c3b80af-9651-40cc-95aa-293d7d0ac560"). InnerVolumeSpecName "kube-api-access-464j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:20.818645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.818603    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-464j8\" (UniqueName: \"kubernetes.io/projected/3c3b80af-9651-40cc-95aa-293d7d0ac560-kube-api-access-464j8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.818645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.818639    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3b80af-9651-40cc-95aa-293d7d0ac560-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.818932 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:20.818654    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3b80af-9651-40cc-95aa-293d7d0ac560-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:21.498359 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.498323    5967 generic.go:356] "Generic (PLEG): container finished" podID="1e9d2402-6b56-49de-b2de-8f52f5ced372" containerID="1d8eb62243b9e22b3e3c8507cce6e3c03d96e87929a3605f7ff6b6e6d27b2a7c" exitCode=0
Oct 30 20:57:21.498517 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.498402    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw" event={"ID":"1e9d2402-6b56-49de-b2de-8f52f5ced372","Type":"ContainerDied","Data":"1d8eb62243b9e22b3e3c8507cce6e3c03d96e87929a3605f7ff6b6e6d27b2a7c"}
Oct 30 20:57:21.499868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.499844    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n" event={"ID":"3c3b80af-9651-40cc-95aa-293d7d0ac560","Type":"ContainerDied","Data":"1d8d2aa21148a8a962acaa3b8e0e2398ef5d17e09456bc75b3ea0f31b4d4f9a9"}
Oct 30 20:57:21.499868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.499865    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"
Oct 30 20:57:21.500026 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.499880    5967 scope.go:122] "RemoveContainer" containerID="7a7b4c6a7ff873103f80594034bf105f88560fbf6deb1c0b8dc4f6e60c91dbed"
Oct 30 20:57:21.660676 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.660603    5967 scope.go:122] "RemoveContainer" containerID="e268aee4af61fb57c353256fc06db368374cb13105270c3a7ec8fe825006f71e"
Oct 30 20:57:21.722587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.722562    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"]
Oct 30 20:57:21.728298 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:21.728277    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/distinct-w1-5f9b95-kserve-7ff8984cd6-xq56n"]
Oct 30 20:57:22.656475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.656452    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:22.740839 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.740817    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw\" (UniqueName: \"kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw\") pod \"1e9d2402-6b56-49de-b2de-8f52f5ced372\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") "
Oct 30 20:57:22.741273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.741018    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca\") pod \"1e9d2402-6b56-49de-b2de-8f52f5ced372\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") "
Oct 30 20:57:22.741273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.741047    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca\") pod \"1e9d2402-6b56-49de-b2de-8f52f5ced372\" (UID: \"1e9d2402-6b56-49de-b2de-8f52f5ced372\") "
Oct 30 20:57:22.741584 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.741489    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca" pod "1e9d2402-6b56-49de-b2de-8f52f5ced372" (UID: "1e9d2402-6b56-49de-b2de-8f52f5ced372"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:22.743048 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.743027    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw" pod "1e9d2402-6b56-49de-b2de-8f52f5ced372" (UID: "1e9d2402-6b56-49de-b2de-8f52f5ced372"). InnerVolumeSpecName "kube-api-access-f9xtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:22.743125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.743086    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca" pod "1e9d2402-6b56-49de-b2de-8f52f5ced372" (UID: "1e9d2402-6b56-49de-b2de-8f52f5ced372"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:22.842686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.842620    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xtw\" (UniqueName: \"kubernetes.io/projected/1e9d2402-6b56-49de-b2de-8f52f5ced372-kube-api-access-f9xtw\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:22.842686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.842646    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:22.842686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:22.842657    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d2402-6b56-49de-b2de-8f52f5ced372-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.100899 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.100821    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"]
Oct 30 20:57:23.101481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.101463    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main"
Oct 30 20:57:23.101556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.101482    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main"
Oct 30 20:57:23.101556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.101512    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1e9d2402-6b56-49de-b2de-8f52f5ced372" containerName="revoke-keys"
Oct 30 20:57:23.101556 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.101518    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9d2402-6b56-49de-b2de-8f52f5ced372" containerName="revoke-keys"
Oct 30 20:57:23.101659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.101613    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" containerName="main"
Oct 30 20:57:23.101659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.101626    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9d2402-6b56-49de-b2de-8f52f5ced372" containerName="revoke-keys"
Oct 30 20:57:23.105164 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.105142    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.162833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.162809    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"]
Oct 30 20:57:23.248084 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.248055    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh67\" (UniqueName: \"kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.248231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.248098    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.248231 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.248175    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.348972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.348943    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh67\" (UniqueName: \"kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.349107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.348982    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.349107 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.349032    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.349653 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.349635    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.351391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.351341    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.406763 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.406722    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh67\" (UniqueName: \"kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67\") pod \"maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.419614 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.419594    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:23.516228 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.516198    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw" event={"ID":"1e9d2402-6b56-49de-b2de-8f52f5ced372","Type":"ContainerDied","Data":"5ff8b14ef05045671bacd28bd5766623c4280387540f7d9dc8b7cdea9b148c96"}
Oct 30 20:57:23.516400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.516250    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff8b14ef05045671bacd28bd5766623c4280387540f7d9dc8b7cdea9b148c96"
Oct 30 20:57:23.516400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.516305    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"
Oct 30 20:57:23.544229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.544194    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3b80af-9651-40cc-95aa-293d7d0ac560" path="/var/lib/kubelet/pods/3c3b80af-9651-40cc-95aa-293d7d0ac560/volumes"
Oct 30 20:57:23.714501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.714479    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"]
Oct 30 20:57:23.818371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:23.818344    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"]
Oct 30 20:57:23.835161 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:57:23.835135    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab35a934_39fa_44db_b477_d6a7172dc3ea.slice/crio-d9f2a07505aab603f3dba785458e6c9293cbda0f3856cd92005823c9a1ef90b4 WatchSource:0}: Error finding container d9f2a07505aab603f3dba785458e6c9293cbda0f3856cd92005823c9a1ef90b4: Status 404 returned error can't find the container with id d9f2a07505aab603f3dba785458e6c9293cbda0f3856cd92005823c9a1ef90b4
Oct 30 20:57:24.524467 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:24.524432    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd" event={"ID":"ab35a934-39fa-44db-b477-d6a7172dc3ea","Type":"ContainerStarted","Data":"3ee53b4a6e11376c721aca7e26df4665f4f6972db7fb7f0d5b8bd672b0bced28"}
Oct 30 20:57:24.524644 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:24.524490    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd" event={"ID":"ab35a934-39fa-44db-b477-d6a7172dc3ea","Type":"ContainerStarted","Data":"d9f2a07505aab603f3dba785458e6c9293cbda0f3856cd92005823c9a1ef90b4"}
Oct 30 20:57:24.654813 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:24.654728    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd" podStartSLOduration=1.654715755 podStartE2EDuration="1.654715755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:24.645717623 +0000 UTC m=+2225.246519583" watchObservedRunningTime="2026-10-30 20:57:24.654715755 +0000 UTC m=+2225.255517714"
Oct 30 20:57:25.534008 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:25.533978    5967 generic.go:356] "Generic (PLEG): container finished" podID="ab35a934-39fa-44db-b477-d6a7172dc3ea" containerID="3ee53b4a6e11376c721aca7e26df4665f4f6972db7fb7f0d5b8bd672b0bced28" exitCode=0
Oct 30 20:57:25.534453 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:25.534042    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd" event={"ID":"ab35a934-39fa-44db-b477-d6a7172dc3ea","Type":"ContainerDied","Data":"3ee53b4a6e11376c721aca7e26df4665f4f6972db7fb7f0d5b8bd672b0bced28"}
Oct 30 20:57:26.002768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.000198    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"]
Oct 30 20:57:26.004648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.004620    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27219c73-nrv5r"]
Oct 30 20:57:26.683515 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.683492    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:26.804076 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.803982    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67\" (UniqueName: \"kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67\") pod \"ab35a934-39fa-44db-b477-d6a7172dc3ea\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") "
Oct 30 20:57:26.804076 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.804053    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca\") pod \"ab35a934-39fa-44db-b477-d6a7172dc3ea\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") "
Oct 30 20:57:26.804321 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.804109    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca\") pod \"ab35a934-39fa-44db-b477-d6a7172dc3ea\" (UID: \"ab35a934-39fa-44db-b477-d6a7172dc3ea\") "
Oct 30 20:57:26.804645 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.804592    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca" pod "ab35a934-39fa-44db-b477-d6a7172dc3ea" (UID: "ab35a934-39fa-44db-b477-d6a7172dc3ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:26.805318 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.804919    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:26.806896 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.806868    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67" pod "ab35a934-39fa-44db-b477-d6a7172dc3ea" (UID: "ab35a934-39fa-44db-b477-d6a7172dc3ea"). InnerVolumeSpecName "kube-api-access-sxh67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:26.807293 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.807260    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca" pod "ab35a934-39fa-44db-b477-d6a7172dc3ea" (UID: "ab35a934-39fa-44db-b477-d6a7172dc3ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:26.906696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.906666    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxh67\" (UniqueName: \"kubernetes.io/projected/ab35a934-39fa-44db-b477-d6a7172dc3ea-kube-api-access-sxh67\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:26.906696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:26.906693    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab35a934-39fa-44db-b477-d6a7172dc3ea-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:27.544013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:27.543981    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3db7ff-db4d-4840-a24b-b02b26ae182b" path="/var/lib/kubelet/pods/9d3db7ff-db4d-4840-a24b-b02b26ae182b/volumes"
Oct 30 20:57:27.548552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:27.548527    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd" event={"ID":"ab35a934-39fa-44db-b477-d6a7172dc3ea","Type":"ContainerDied","Data":"d9f2a07505aab603f3dba785458e6c9293cbda0f3856cd92005823c9a1ef90b4"}
Oct 30 20:57:27.548702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:27.548569    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f2a07505aab603f3dba785458e6c9293cbda0f3856cd92005823c9a1ef90b4"
Oct 30 20:57:27.548702 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:27.548538    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"
Oct 30 20:57:29.047700 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:29.047670    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"]
Oct 30 20:57:29.096811 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:29.096781    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2486e8df-m95xb"]
Oct 30 20:57:29.545652 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:29.545616    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ac5c9f-5f94-4f12-81c9-97b0964c532d" path="/var/lib/kubelet/pods/92ac5c9f-5f94-4f12-81c9-97b0964c532d/volumes"
Oct 30 20:57:30.621344 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:30.621314    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:30.621713 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:30.621315    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:57:31.980787 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:31.980724    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:57:32.709762 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:32.709713    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"]
Oct 30 20:57:32.709934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:32.709912    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" containerID="cri-o://412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a" gracePeriod=30
Oct 30 20:57:32.788162 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:32.788087    5967 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:57:32.788347 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:32.788204    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:57:35.185200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.185164    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"]
Oct 30 20:57:35.185772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.185758    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ab35a934-39fa-44db-b477-d6a7172dc3ea" containerName="revoke-keys"
Oct 30 20:57:35.185829 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.185772    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab35a934-39fa-44db-b477-d6a7172dc3ea" containerName="revoke-keys"
Oct 30 20:57:35.185911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.185902    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab35a934-39fa-44db-b477-d6a7172dc3ea" containerName="revoke-keys"
Oct 30 20:57:35.190607 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.190580    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.213471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.213445    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsgs\" (UniqueName: \"kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.213593 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.213486    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.213593 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.213504    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.228119 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.228097    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"]
Oct 30 20:57:35.231984 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.231969    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.247969 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.247944    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"]
Oct 30 20:57:35.278346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.278313    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:57:35.278638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.278615    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:35.299308 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.297917    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:57:35.309986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.309905    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:57:35.318257 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.316874    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"]
Oct 30 20:57:35.318616 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.318168    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.318760 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.318715    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftz49\" (UniqueName: \"kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.318992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.318961    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsgs\" (UniqueName: \"kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.319125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.319019    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.319125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.319090    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.319248 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.319126    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.320216 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.320180    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.322975 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.322954    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.375495 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.375462    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsgs\" (UniqueName: \"kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs\") pod \"maas-api-revoke-sub-91a748b0-f6qkt\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.428428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.420601    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.428428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.420677    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.428428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.420781    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftz49\" (UniqueName: \"kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.428428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.421918    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.432186 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.432158    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.477438 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.477412    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftz49\" (UniqueName: \"kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49\") pod \"maas-api-revoke-sub-2d6d1a0f-nb7qc\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.504012 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.503990    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:35.544134 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.544106    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:35.882081 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.882053    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"]
Oct 30 20:57:35.883641 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:57:35.883616    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad63ea7_b252_470b_9ab8_65323de7bdd2.slice/crio-b35734936e7d16f31a188c492c9c38e72137eb3c4a1dc6630ed156e794dc4123 WatchSource:0}: Error finding container b35734936e7d16f31a188c492c9c38e72137eb3c4a1dc6630ed156e794dc4123: Status 404 returned error can't find the container with id b35734936e7d16f31a188c492c9c38e72137eb3c4a1dc6630ed156e794dc4123
Oct 30 20:57:35.933075 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:35.933051    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"]
Oct 30 20:57:35.933814 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:57:35.933779    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda104cceb_0668_438b_a90f_b64b75162f68.slice/crio-51f517d45dfcd8306cf3059303c3c698efca5afcc3be87241e2c4e94f3480175 WatchSource:0}: Error finding container 51f517d45dfcd8306cf3059303c3c698efca5afcc3be87241e2c4e94f3480175: Status 404 returned error can't find the container with id 51f517d45dfcd8306cf3059303c3c698efca5afcc3be87241e2c4e94f3480175
Oct 30 20:57:36.622300 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:36.622263    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc" event={"ID":"a104cceb-0668-438b-a90f-b64b75162f68","Type":"ContainerStarted","Data":"1f49c970f3ad55d22c5ee74023b207f0bd643c822de588923e5b5f6b8d2eb872"}
Oct 30 20:57:36.622824 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:36.622326    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc" event={"ID":"a104cceb-0668-438b-a90f-b64b75162f68","Type":"ContainerStarted","Data":"51f517d45dfcd8306cf3059303c3c698efca5afcc3be87241e2c4e94f3480175"}
Oct 30 20:57:36.624853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:36.624771    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt" event={"ID":"fad63ea7-b252-470b-9ab8-65323de7bdd2","Type":"ContainerStarted","Data":"75e58682ba2056c6d20565a55cbea508350bc95bd39507c3fa3cdb366cf29e55"}
Oct 30 20:57:36.624853 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:36.624822    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt" event={"ID":"fad63ea7-b252-470b-9ab8-65323de7bdd2","Type":"ContainerStarted","Data":"b35734936e7d16f31a188c492c9c38e72137eb3c4a1dc6630ed156e794dc4123"}
Oct 30 20:57:36.739236 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:36.739189    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc" podStartSLOduration=1.739175782 podStartE2EDuration="1.739175782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:36.735655466 +0000 UTC m=+2237.336457449" watchObservedRunningTime="2026-10-30 20:57:36.739175782 +0000 UTC m=+2237.339977742"
Oct 30 20:57:36.851830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:36.851786    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt" podStartSLOduration=1.8517720930000001 podStartE2EDuration="1.851772093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:36.848475195 +0000 UTC m=+2237.449277155" watchObservedRunningTime="2026-10-30 20:57:36.851772093 +0000 UTC m=+2237.452574052"
Oct 30 20:57:37.645772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.643274    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:57:37.687862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.648924    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume\") pod \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") "
Oct 30 20:57:37.687862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.649096    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn\" (UniqueName: \"kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn\") pod \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\" (UID: \"2bd52691-dd57-4252-a51d-3b1631bcf8a4\") "
Oct 30 20:57:37.687862 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.679473    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume" pod "2bd52691-dd57-4252-a51d-3b1631bcf8a4" (UID: "2bd52691-dd57-4252-a51d-3b1631bcf8a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:37.714970 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.714582    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd52691-dd57-4252-a51d-3b1631bcf8a4-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:37.738830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.733008    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn" pod "2bd52691-dd57-4252-a51d-3b1631bcf8a4" (UID: "2bd52691-dd57-4252-a51d-3b1631bcf8a4"). InnerVolumeSpecName "kube-api-access-9gxmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:37.738830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.734726    5967 generic.go:356] "Generic (PLEG): container finished" podID="fad63ea7-b252-470b-9ab8-65323de7bdd2" containerID="75e58682ba2056c6d20565a55cbea508350bc95bd39507c3fa3cdb366cf29e55" exitCode=0
Oct 30 20:57:37.738830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.734844    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt" event={"ID":"fad63ea7-b252-470b-9ab8-65323de7bdd2","Type":"ContainerDied","Data":"75e58682ba2056c6d20565a55cbea508350bc95bd39507c3fa3cdb366cf29e55"}
Oct 30 20:57:37.738830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.737377    5967 generic.go:356] "Generic (PLEG): container finished" podID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerID="0435a2d9331ee04443a37260ef73594bd77db7246513fa880dcbe98ec3c8417a" exitCode=0
Oct 30 20:57:37.738830 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.737525    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" event={"ID":"59a64963-39b1-4577-b18e-50d8b4fa98c9","Type":"ContainerDied","Data":"0435a2d9331ee04443a37260ef73594bd77db7246513fa880dcbe98ec3c8417a"}
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.740219    5967 generic.go:356] "Generic (PLEG): container finished" podID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerID="9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f" exitCode=0
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.740294    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" event={"ID":"2bd52691-dd57-4252-a51d-3b1631bcf8a4","Type":"ContainerDied","Data":"9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f"}
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.740334    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb" event={"ID":"2bd52691-dd57-4252-a51d-3b1631bcf8a4","Type":"ContainerDied","Data":"f2b0ab65311f61b51d3e1aa34a532154c668018a460d53929db02041020ee3ea"}
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.740362    5967 scope.go:122] "RemoveContainer" containerID="9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f"
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.740515    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.753215    5967 generic.go:356] "Generic (PLEG): container finished" podID="a104cceb-0668-438b-a90f-b64b75162f68" containerID="1f49c970f3ad55d22c5ee74023b207f0bd643c822de588923e5b5f6b8d2eb872" exitCode=0
Oct 30 20:57:37.761572 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.753385    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc" event={"ID":"a104cceb-0668-438b-a90f-b64b75162f68","Type":"ContainerDied","Data":"1f49c970f3ad55d22c5ee74023b207f0bd643c822de588923e5b5f6b8d2eb872"}
Oct 30 20:57:37.773217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.773194    5967 scope.go:122] "RemoveContainer" containerID="9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f"
Oct 30 20:57:37.773689 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:37.773660    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f\": container with ID starting with 9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f not found: ID does not exist" containerID="9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f"
Oct 30 20:57:37.773807 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.773699    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f"} err="failed to get container status \"9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f\": rpc error: code = NotFound desc = could not find container \"9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f\": container with ID starting with 9906e39bf508faf50b39402dc1567710274140dfcf11eefe17edaad1db0a227f not found: ID does not exist"
Oct 30 20:57:37.800815 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.800785    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:57:37.815433 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.815411    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gxmn\" (UniqueName: \"kubernetes.io/projected/2bd52691-dd57-4252-a51d-3b1631bcf8a4-kube-api-access-9gxmn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:37.895876 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.895778    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"]
Oct 30 20:57:37.903481 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.903451    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb"]
Oct 30 20:57:37.916531 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.916507    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume\") pod \"59a64963-39b1-4577-b18e-50d8b4fa98c9\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") "
Oct 30 20:57:37.916669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.916567    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt\" (UniqueName: \"kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt\") pod \"59a64963-39b1-4577-b18e-50d8b4fa98c9\" (UID: \"59a64963-39b1-4577-b18e-50d8b4fa98c9\") "
Oct 30 20:57:37.916970 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.916945    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume" pod "59a64963-39b1-4577-b18e-50d8b4fa98c9" (UID: "59a64963-39b1-4577-b18e-50d8b4fa98c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:37.917241 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.917225    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a64963-39b1-4577-b18e-50d8b4fa98c9-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:37.919371 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:37.919345    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt" pod "59a64963-39b1-4577-b18e-50d8b4fa98c9" (UID: "59a64963-39b1-4577-b18e-50d8b4fa98c9"). InnerVolumeSpecName "kube-api-access-vx2gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:38.019200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.019171    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2gt\" (UniqueName: \"kubernetes.io/projected/59a64963-39b1-4577-b18e-50d8b4fa98c9-kube-api-access-vx2gt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.028217 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.028196    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:57:38.120567 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120535    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.120869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120618    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.120869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120664    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.120869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120702    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.120869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120784    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.121105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120881    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.121105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120909    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.121105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120931    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.121105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.120972    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.121105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121050    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.121105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121104    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k\") pod \"8e1ba812-68ce-4511-b06b-ce016de4b110\" (UID: \"8e1ba812-68ce-4511-b06b-ce016de4b110\") "
Oct 30 20:57:38.121389 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121198    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:38.121449 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121389    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.121507 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121470    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.121884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121861    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.122154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121888    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.122154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121904    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ba812-68ce-4511-b06b-ce016de4b110-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.122154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121917    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.122154 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.121931    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.123337 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.123313    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:38.123489 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.123464    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:38.124277 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.124261    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.124340 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.124307    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k" pod "8e1ba812-68ce-4511-b06b-ce016de4b110" (UID: "8e1ba812-68ce-4511-b06b-ce016de4b110"). InnerVolumeSpecName "kube-api-access-rsn6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:38.222867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.222835    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.222867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.222862    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.222867 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.222871    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ba812-68ce-4511-b06b-ce016de4b110-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.223085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.222880    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsn6k\" (UniqueName: \"kubernetes.io/projected/8e1ba812-68ce-4511-b06b-ce016de4b110-kube-api-access-rsn6k\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.735666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.735609    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main" containerID="cri-o://f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94" gracePeriod=45
Oct 30 20:57:38.763418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.763388    5967 generic.go:356] "Generic (PLEG): container finished" podID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerID="412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a" exitCode=0
Oct 30 20:57:38.763562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.763453    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"
Oct 30 20:57:38.763562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.763463    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" event={"ID":"8e1ba812-68ce-4511-b06b-ce016de4b110","Type":"ContainerDied","Data":"412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a"}
Oct 30 20:57:38.763562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.763519    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt" event={"ID":"8e1ba812-68ce-4511-b06b-ce016de4b110","Type":"ContainerDied","Data":"33751758dbfd4a187de8e2e205b56a12a4dba9679da10cbbf03129227b78e688"}
Oct 30 20:57:38.763562 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.763543    5967 scope.go:122] "RemoveContainer" containerID="412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a"
Oct 30 20:57:38.765169 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.765150    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"
Oct 30 20:57:38.765264 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.765150    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn" event={"ID":"59a64963-39b1-4577-b18e-50d8b4fa98c9","Type":"ContainerDied","Data":"a2702b53bfb1368df3cb5122c8dec6e77f8c7a10dfaf893047ffaefb349feb3d"}
Oct 30 20:57:38.854259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.853946    5967 scope.go:122] "RemoveContainer" containerID="412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a"
Oct 30 20:57:38.854434 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:38.854291    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a\": container with ID starting with 412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a not found: ID does not exist" containerID="412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a"
Oct 30 20:57:38.854434 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.854325    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a"} err="failed to get container status \"412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a\": rpc error: code = NotFound desc = could not find container \"412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a\": container with ID starting with 412431283f771384452c23e2d43f19e1a0fac167cff515ef903a8a1df9bdf48a not found: ID does not exist"
Oct 30 20:57:38.854434 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.854354    5967 scope.go:122] "RemoveContainer" containerID="0435a2d9331ee04443a37260ef73594bd77db7246513fa880dcbe98ec3c8417a"
Oct 30 20:57:38.964359 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.964337    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:38.967200 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.967177    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:38.982682 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:38.982651    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"]
Oct 30 20:57:39.032658 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.032584    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt"]
Oct 30 20:57:39.055619 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.055588    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:57:39.134473 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134445    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca\") pod \"a104cceb-0668-438b-a90f-b64b75162f68\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") "
Oct 30 20:57:39.134650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134520    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca\") pod \"a104cceb-0668-438b-a90f-b64b75162f68\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") "
Oct 30 20:57:39.134650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134551    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49\" (UniqueName: \"kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49\") pod \"a104cceb-0668-438b-a90f-b64b75162f68\" (UID: \"a104cceb-0668-438b-a90f-b64b75162f68\") "
Oct 30 20:57:39.134650 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134574    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca\") pod \"fad63ea7-b252-470b-9ab8-65323de7bdd2\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") "
Oct 30 20:57:39.134854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134671    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm\") pod \"daf85b46-827c-4f06-a493-e7c47c1c3681\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") "
Oct 30 20:57:39.134854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134715    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir\") pod \"daf85b46-827c-4f06-a493-e7c47c1c3681\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") "
Oct 30 20:57:39.135018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134874    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache\") pod \"daf85b46-827c-4f06-a493-e7c47c1c3681\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") "
Oct 30 20:57:39.135018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134950    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs\" (UniqueName: \"kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs\") pod \"daf85b46-827c-4f06-a493-e7c47c1c3681\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") "
Oct 30 20:57:39.135018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.134992    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs\" (UniqueName: \"kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs\") pod \"fad63ea7-b252-470b-9ab8-65323de7bdd2\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") "
Oct 30 20:57:39.135018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.135001    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir" pod "daf85b46-827c-4f06-a493-e7c47c1c3681" (UID: "daf85b46-827c-4f06-a493-e7c47c1c3681"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.135018 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.135007    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca" pod "a104cceb-0668-438b-a90f-b64b75162f68" (UID: "a104cceb-0668-438b-a90f-b64b75162f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:39.135291 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.135227    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache" pod "daf85b46-827c-4f06-a493-e7c47c1c3681" (UID: "daf85b46-827c-4f06-a493-e7c47c1c3681"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.135355 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.135295    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca" pod "fad63ea7-b252-470b-9ab8-65323de7bdd2" (UID: "fad63ea7-b252-470b-9ab8-65323de7bdd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:39.135638 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.135568    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp\" (UniqueName: \"kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp\") pod \"daf85b46-827c-4f06-a493-e7c47c1c3681\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") "
Oct 30 20:57:39.136501 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.136473    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca\") pod \"fad63ea7-b252-470b-9ab8-65323de7bdd2\" (UID: \"fad63ea7-b252-470b-9ab8-65323de7bdd2\") "
Oct 30 20:57:39.136611 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.136557    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home\") pod \"daf85b46-827c-4f06-a493-e7c47c1c3681\" (UID: \"daf85b46-827c-4f06-a493-e7c47c1c3681\") "
Oct 30 20:57:39.137219 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.137192    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.137310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.137223    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.137310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.137238    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.137310 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.137250    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.137548 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.137523    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home" pod "daf85b46-827c-4f06-a493-e7c47c1c3681" (UID: "daf85b46-827c-4f06-a493-e7c47c1c3681"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.138145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.138078    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs" pod "daf85b46-827c-4f06-a493-e7c47c1c3681" (UID: "daf85b46-827c-4f06-a493-e7c47c1c3681"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:39.138363 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.138226    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49" pod "a104cceb-0668-438b-a90f-b64b75162f68" (UID: "a104cceb-0668-438b-a90f-b64b75162f68"). InnerVolumeSpecName "kube-api-access-ftz49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.142011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.141872    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca" pod "a104cceb-0668-438b-a90f-b64b75162f68" (UID: "a104cceb-0668-438b-a90f-b64b75162f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:39.142011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.141902    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs" pod "fad63ea7-b252-470b-9ab8-65323de7bdd2" (UID: "fad63ea7-b252-470b-9ab8-65323de7bdd2"). InnerVolumeSpecName "kube-api-access-whsgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.142011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.141909    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca" pod "fad63ea7-b252-470b-9ab8-65323de7bdd2" (UID: "fad63ea7-b252-470b-9ab8-65323de7bdd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:39.142011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.141979    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp" pod "daf85b46-827c-4f06-a493-e7c47c1c3681" (UID: "daf85b46-827c-4f06-a493-e7c47c1c3681"). InnerVolumeSpecName "kube-api-access-q8btp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.142329 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.142210    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm" pod "daf85b46-827c-4f06-a493-e7c47c1c3681" (UID: "daf85b46-827c-4f06-a493-e7c47c1c3681"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:39.145400 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.145376    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"]
Oct 30 20:57:39.146551 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.146537    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn"]
Oct 30 20:57:39.238356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238324    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad63ea7-b252-470b-9ab8-65323de7bdd2-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238353    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238356 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238366    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a104cceb-0668-438b-a90f-b64b75162f68-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238376    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftz49\" (UniqueName: \"kubernetes.io/projected/a104cceb-0668-438b-a90f-b64b75162f68-kube-api-access-ftz49\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238384    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf85b46-827c-4f06-a493-e7c47c1c3681-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238392    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf85b46-827c-4f06-a493-e7c47c1c3681-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238400    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whsgs\" (UniqueName: \"kubernetes.io/projected/fad63ea7-b252-470b-9ab8-65323de7bdd2-kube-api-access-whsgs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.238667 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.238408    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8btp\" (UniqueName: \"kubernetes.io/projected/daf85b46-827c-4f06-a493-e7c47c1c3681-kube-api-access-q8btp\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.543059 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.543027    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" path="/var/lib/kubelet/pods/2bd52691-dd57-4252-a51d-3b1631bcf8a4/volumes"
Oct 30 20:57:39.543471 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.543451    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" path="/var/lib/kubelet/pods/59a64963-39b1-4577-b18e-50d8b4fa98c9/volumes"
Oct 30 20:57:39.543936 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.543918    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" path="/var/lib/kubelet/pods/8e1ba812-68ce-4511-b06b-ce016de4b110/volumes"
Oct 30 20:57:39.773287 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.773263    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"
Oct 30 20:57:39.773677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.773305    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc" event={"ID":"a104cceb-0668-438b-a90f-b64b75162f68","Type":"ContainerDied","Data":"51f517d45dfcd8306cf3059303c3c698efca5afcc3be87241e2c4e94f3480175"}
Oct 30 20:57:39.773677 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.773353    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51f517d45dfcd8306cf3059303c3c698efca5afcc3be87241e2c4e94f3480175"
Oct 30 20:57:39.774660 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.774632    5967 generic.go:356] "Generic (PLEG): container finished" podID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerID="f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94" exitCode=0
Oct 30 20:57:39.774810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.774671    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"
Oct 30 20:57:39.774810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.774714    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" event={"ID":"daf85b46-827c-4f06-a493-e7c47c1c3681","Type":"ContainerDied","Data":"f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94"}
Oct 30 20:57:39.774810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.774775    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8" event={"ID":"daf85b46-827c-4f06-a493-e7c47c1c3681","Type":"ContainerDied","Data":"be2fdd8ba8630618fc64302b6028a5925a91004647a2721302ddcf3bfd080d5b"}
Oct 30 20:57:39.774810 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.774802    5967 scope.go:122] "RemoveContainer" containerID="f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94"
Oct 30 20:57:39.777849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.777822    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt" event={"ID":"fad63ea7-b252-470b-9ab8-65323de7bdd2","Type":"ContainerDied","Data":"b35734936e7d16f31a188c492c9c38e72137eb3c4a1dc6630ed156e794dc4123"}
Oct 30 20:57:39.777966 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.777847    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"
Oct 30 20:57:39.777966 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.777861    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b35734936e7d16f31a188c492c9c38e72137eb3c4a1dc6630ed156e794dc4123"
Oct 30 20:57:39.785506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.785488    5967 scope.go:122] "RemoveContainer" containerID="f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94"
Oct 30 20:57:39.785774 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:57:39.785756    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94\": container with ID starting with f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94 not found: ID does not exist" containerID="f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94"
Oct 30 20:57:39.785831 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.785781    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94"} err="failed to get container status \"f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94\": rpc error: code = NotFound desc = could not find container \"f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94\": container with ID starting with f671546fe2e5dec57ca804d34c7dbd5423a6437d92f251ed201588e3ac96fc94 not found: ID does not exist"
Oct 30 20:57:39.937785 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.937762    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"]
Oct 30 20:57:39.941197 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:39.941173    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/simulator-w5-a78bf4-kserve-848d9f5dd8-gtck8"]
Oct 30 20:57:41.542709 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:41.542676    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" path="/var/lib/kubelet/pods/daf85b46-827c-4f06-a493-e7c47c1c3681/volumes"
Oct 30 20:57:51.845963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.845929    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"]
Oct 30 20:57:51.846492 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846479    5967 cpu_manager.go:414] "containerMap: removing container" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main"
Oct 30 20:57:51.846536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846492    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main"
Oct 30 20:57:51.846536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846510    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerName="payload-processing"
Oct 30 20:57:51.846536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846515    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerName="payload-processing"
Oct 30 20:57:51.846536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846523    5967 cpu_manager.go:414] "containerMap: removing container" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerName="payload-pre-processing"
Oct 30 20:57:51.846536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846528    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerName="payload-pre-processing"
Oct 30 20:57:51.846536 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846537    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a104cceb-0668-438b-a90f-b64b75162f68" containerName="revoke-keys"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846542    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a104cceb-0668-438b-a90f-b64b75162f68" containerName="revoke-keys"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846556    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846560    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846577    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fad63ea7-b252-470b-9ab8-65323de7bdd2" containerName="revoke-keys"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846581    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad63ea7-b252-470b-9ab8-65323de7bdd2" containerName="revoke-keys"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846670    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf85b46-827c-4f06-a493-e7c47c1c3681" containerName="main"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846680    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd52691-dd57-4252-a51d-3b1631bcf8a4" containerName="payload-processing"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846687    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a64963-39b1-4577-b18e-50d8b4fa98c9" containerName="payload-pre-processing"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846693    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a104cceb-0668-438b-a90f-b64b75162f68" containerName="revoke-keys"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846699    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1ba812-68ce-4511-b06b-ce016de4b110" containerName="istio-proxy"
Oct 30 20:57:51.846726 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.846707    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad63ea7-b252-470b-9ab8-65323de7bdd2" containerName="revoke-keys"
Oct 30 20:57:51.850350 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.850334    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.899730 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.899698    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.899930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.899753    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.899930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.899798    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.899930 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.899915    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.900094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.900036    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.900141 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.900098    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.900195 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.900177    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.900305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.900291    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4fwt\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.900366 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.900343    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:51.901954 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.901922    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-23491e-openshift-default-dockercfg-h6dnx\""
Oct 30 20:57:51.955669 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:51.955637    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"]
Oct 30 20:57:52.001497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001465    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.001648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001512    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.001648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001557    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.001648 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001586    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.001845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001691    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.001845 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001823    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.001936 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.001920    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002039 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002017    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4fwt\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002052    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002082    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002174 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002140    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002324    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002427    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.002621 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.002599    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.004190 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.004167    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.004510 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.004492    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.064229 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.064195    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4fwt\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.066583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.066493    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token\") pod \"e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") " pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.164215 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.164137    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:52.596752 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.596709    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"]
Oct 30 20:57:52.598004 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:57:52.597975    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7dfda0_0021_4b45_bc15_044422fad237.slice/crio-999dfe2fc38fa48404e19c83a6d5a3ffca81cca27af2a72d802fe65fd2e2246b WatchSource:0}: Error finding container 999dfe2fc38fa48404e19c83a6d5a3ffca81cca27af2a72d802fe65fd2e2246b: Status 404 returned error can't find the container with id 999dfe2fc38fa48404e19c83a6d5a3ffca81cca27af2a72d802fe65fd2e2246b
Oct 30 20:57:52.600360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.600330    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:52.600428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.600395    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:52.600428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.600424    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:57:52.882819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.882705    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" event={"ID":"0d7dfda0-0021-4b45-bc15-044422fad237","Type":"ContainerStarted","Data":"997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"}
Oct 30 20:57:52.882819 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:52.882779    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" event={"ID":"0d7dfda0-0021-4b45-bc15-044422fad237","Type":"ContainerStarted","Data":"999dfe2fc38fa48404e19c83a6d5a3ffca81cca27af2a72d802fe65fd2e2246b"}
Oct 30 20:57:53.000696 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:53.000451    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podStartSLOduration=2.00043153 podStartE2EDuration="2.00043153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:52.999241861 +0000 UTC m=+2253.600043821" watchObservedRunningTime="2026-10-30 20:57:53.00043153 +0000 UTC m=+2253.601233519"
Oct 30 20:57:53.164852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:53.164757    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:53.171113 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:53.171091    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:53.889493 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:53.889468    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:57:53.890405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:57:53.890388    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:58:01.980345 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:58:01.980264    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:58:19.372068 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.372037    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"]
Oct 30 20:58:19.376091 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.376075    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.430411 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.430387    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"]
Oct 30 20:58:19.434768 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.434730    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.467553 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.467525    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.467952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.467682    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5725\" (UniqueName: \"kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.467952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.467829    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.477934 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.477912    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:58:19.479973 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.479949    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:58:19.480087 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.480027    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:58:19.480087 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.480042    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:58:19.482631 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.482612    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"]
Oct 30 20:58:19.546120 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.546093    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"]
Oct 30 20:58:19.569534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.569510    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5725\" (UniqueName: \"kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.569681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.569604    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.569681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.569635    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.569854 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.569832    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.569905 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.569878    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.569958 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.569902    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jscq\" (UniqueName: \"kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.620549 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.620523    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:58:19.620773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.620721    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:58:19.630772 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.630708    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.632432 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.632413    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.670937 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.670908    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.671067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.671019    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.671067 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.671048    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jscq\" (UniqueName: \"kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.671686 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.671663    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.673439 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.673419    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.674698 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.674678    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:58:19.686182 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.686158    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5725\" (UniqueName: \"kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725\") pod \"maas-api-revoke-sub-daf3e76e-fdmf4\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.727087 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.727069    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jscq\" (UniqueName: \"kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq\") pod \"maas-api-revoke-sub-8549e3cb-zf8rw\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:19.740781 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.740761    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:58:19.750869 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.750851    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:19.750959 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:19.750905    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:20.131096 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:20.131073    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"]
Oct 30 20:58:20.132877 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:58:20.132853    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f859df_b5e7_4821_bb2a_7409463fcdcc.slice/crio-5a4081dc75245a24defe920665ee5f3180115a1406cffd452163b2f9cbac8234 WatchSource:0}: Error finding container 5a4081dc75245a24defe920665ee5f3180115a1406cffd452163b2f9cbac8234: Status 404 returned error can't find the container with id 5a4081dc75245a24defe920665ee5f3180115a1406cffd452163b2f9cbac8234
Oct 30 20:58:20.178663 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:20.178631    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"]
Oct 30 20:58:20.179831 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:58:20.179805    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d296c8_49bc_4d0c_885f_d781dc5ae26b.slice/crio-c2f8e7f3d8f783b3dbab330604326ba03628636237d33d41b2b0b639524d7f3f WatchSource:0}: Error finding container c2f8e7f3d8f783b3dbab330604326ba03628636237d33d41b2b0b639524d7f3f: Status 404 returned error can't find the container with id c2f8e7f3d8f783b3dbab330604326ba03628636237d33d41b2b0b639524d7f3f
Oct 30 20:58:21.083984 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:21.083952    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4" event={"ID":"f4f859df-b5e7-4821-bb2a-7409463fcdcc","Type":"ContainerStarted","Data":"df3783656b3b075bbdfc2a006e124a35a879323347d6ca494403ad3c3b33f8c8"}
Oct 30 20:58:21.084393 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:21.084011    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4" event={"ID":"f4f859df-b5e7-4821-bb2a-7409463fcdcc","Type":"ContainerStarted","Data":"5a4081dc75245a24defe920665ee5f3180115a1406cffd452163b2f9cbac8234"}
Oct 30 20:58:21.085559 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:21.085537    5967 generic.go:356] "Generic (PLEG): container finished" podID="65d296c8-49bc-4d0c-885f-d781dc5ae26b" containerID="c705e278bdfa915d6490077c80b67ab8ecdd3ee58dad713c7b9326e0c5a267a1" exitCode=0
Oct 30 20:58:21.085668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:21.085617    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw" event={"ID":"65d296c8-49bc-4d0c-885f-d781dc5ae26b","Type":"ContainerDied","Data":"c705e278bdfa915d6490077c80b67ab8ecdd3ee58dad713c7b9326e0c5a267a1"}
Oct 30 20:58:21.085668 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:21.085654    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw" event={"ID":"65d296c8-49bc-4d0c-885f-d781dc5ae26b","Type":"ContainerStarted","Data":"c2f8e7f3d8f783b3dbab330604326ba03628636237d33d41b2b0b639524d7f3f"}
Oct 30 20:58:22.093951 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.093911    5967 generic.go:356] "Generic (PLEG): container finished" podID="f4f859df-b5e7-4821-bb2a-7409463fcdcc" containerID="df3783656b3b075bbdfc2a006e124a35a879323347d6ca494403ad3c3b33f8c8" exitCode=0
Oct 30 20:58:22.094426 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.093996    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4" event={"ID":"f4f859df-b5e7-4821-bb2a-7409463fcdcc","Type":"ContainerDied","Data":"df3783656b3b075bbdfc2a006e124a35a879323347d6ca494403ad3c3b33f8c8"}
Oct 30 20:58:22.256135 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.256107    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:22.282749 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.282713    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:22.409078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.408994    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca\") pod \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") "
Oct 30 20:58:22.409078 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.409038    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca\") pod \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") "
Oct 30 20:58:22.409313 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.409231    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca\") pod \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") "
Oct 30 20:58:22.409428 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.409404    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq\" (UniqueName: \"kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq\") pod \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") "
Oct 30 20:58:22.409490 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.409460    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca\") pod \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\" (UID: \"65d296c8-49bc-4d0c-885f-d781dc5ae26b\") "
Oct 30 20:58:22.409552 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.409511    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725\" (UniqueName: \"kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725\") pod \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\" (UID: \"f4f859df-b5e7-4821-bb2a-7409463fcdcc\") "
Oct 30 20:58:22.409634 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.409608    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca" pod "f4f859df-b5e7-4821-bb2a-7409463fcdcc" (UID: "f4f859df-b5e7-4821-bb2a-7409463fcdcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:22.410360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.410140    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca" pod "65d296c8-49bc-4d0c-885f-d781dc5ae26b" (UID: "65d296c8-49bc-4d0c-885f-d781dc5ae26b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:22.410360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.410316    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:22.410360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.410336    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:22.411447 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.411410    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca" pod "65d296c8-49bc-4d0c-885f-d781dc5ae26b" (UID: "65d296c8-49bc-4d0c-885f-d781dc5ae26b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:22.411594 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.411499    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca" pod "f4f859df-b5e7-4821-bb2a-7409463fcdcc" (UID: "f4f859df-b5e7-4821-bb2a-7409463fcdcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:22.411946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.411926    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq" pod "65d296c8-49bc-4d0c-885f-d781dc5ae26b" (UID: "65d296c8-49bc-4d0c-885f-d781dc5ae26b"). InnerVolumeSpecName "kube-api-access-4jscq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:22.412414 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.412396    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725" pod "f4f859df-b5e7-4821-bb2a-7409463fcdcc" (UID: "f4f859df-b5e7-4821-bb2a-7409463fcdcc"). InnerVolumeSpecName "kube-api-access-p5725". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:22.511790 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.511763    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jscq\" (UniqueName: \"kubernetes.io/projected/65d296c8-49bc-4d0c-885f-d781dc5ae26b-kube-api-access-4jscq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:22.511925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.511794    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5725\" (UniqueName: \"kubernetes.io/projected/f4f859df-b5e7-4821-bb2a-7409463fcdcc-kube-api-access-p5725\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:22.511925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.511811    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d296c8-49bc-4d0c-885f-d781dc5ae26b-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:22.511925 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:22.511825    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f859df-b5e7-4821-bb2a-7409463fcdcc-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:23.102613 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:23.102584    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"
Oct 30 20:58:23.103050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:23.102621    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw" event={"ID":"65d296c8-49bc-4d0c-885f-d781dc5ae26b","Type":"ContainerDied","Data":"c2f8e7f3d8f783b3dbab330604326ba03628636237d33d41b2b0b639524d7f3f"}
Oct 30 20:58:23.103050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:23.102675    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2f8e7f3d8f783b3dbab330604326ba03628636237d33d41b2b0b639524d7f3f"
Oct 30 20:58:23.104125 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:23.104103    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"
Oct 30 20:58:23.104256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:23.104102    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4" event={"ID":"f4f859df-b5e7-4821-bb2a-7409463fcdcc","Type":"ContainerDied","Data":"5a4081dc75245a24defe920665ee5f3180115a1406cffd452163b2f9cbac8234"}
Oct 30 20:58:23.104256 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:23.104227    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a4081dc75245a24defe920665ee5f3180115a1406cffd452163b2f9cbac8234"
Oct 30 20:58:31.979948 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:58:31.979912    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:58:36.879315 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:36.879283    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"]
Oct 30 20:58:36.926774 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:36.926726    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f9b95-cw46k"]
Oct 30 20:58:37.547259 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:37.547197    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0738015-c327-417a-ba2f-a371dd5ff212" path="/var/lib/kubelet/pods/e0738015-c327-417a-ba2f-a371dd5ff212/volumes"
Oct 30 20:58:47.267863 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:47.267827    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"]
Oct 30 20:58:47.268370 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:47.268019    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy" containerID="cri-o://a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f" gracePeriod=30
Oct 30 20:58:52.531179 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.531121    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:58:52.623369 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623345    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623388    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623409    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623438    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623454    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623529 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623518    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623559    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623622    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623646    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:52.623894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623660    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:52.623894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623712    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token\") pod \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\" (UID: \"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d\") "
Oct 30 20:58:52.623894 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623805    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:52.624194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.623920    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:52.624194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.624081    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:52.624311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.624239    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.624311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.624255    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.624311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.624269    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.624311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.624282    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.624311 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.624291    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.626593 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.626572    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:58:52.626701 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.626623    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:52.626701 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.626663    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:52.626701 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.626688    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g" pod "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" (UID: "ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d"). InnerVolumeSpecName "kube-api-access-gz79g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:52.725953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.725919    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.725953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.725943    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.725953 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.725953    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz79g\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-kube-api-access-gz79g\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.726170 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:52.725967    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:53.322803 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.322770    5967 generic.go:356] "Generic (PLEG): container finished" podID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerID="a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f" exitCode=0
Oct 30 20:58:53.323016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.322826    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"
Oct 30 20:58:53.323016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.322852    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" event={"ID":"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d","Type":"ContainerDied","Data":"a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f"}
Oct 30 20:58:53.323016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.322939    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz" event={"ID":"ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d","Type":"ContainerDied","Data":"f0ca1176e6a1c9a7d3a35b832d7253fbbbe861b9f0fe56d3e8ad6b988be7db44"}
Oct 30 20:58:53.323016 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.322969    5967 scope.go:122] "RemoveContainer" containerID="a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f"
Oct 30 20:58:53.334659 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.334638    5967 scope.go:122] "RemoveContainer" containerID="14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"
Oct 30 20:58:53.343589 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.343572    5967 scope.go:122] "RemoveContainer" containerID="a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f"
Oct 30 20:58:53.343906 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:58:53.343883    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f\": container with ID starting with a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f not found: ID does not exist" containerID="a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f"
Oct 30 20:58:53.343991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.343917    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f"} err="failed to get container status \"a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f\": rpc error: code = NotFound desc = could not find container \"a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f\": container with ID starting with a70a53624c0c1d034e4bedc570f713a72272ac2561675da5cb12ee5439b9328f not found: ID does not exist"
Oct 30 20:58:53.343991 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.343941    5967 scope.go:122] "RemoveContainer" containerID="14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"
Oct 30 20:58:53.344182 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:58:53.344162    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c\": container with ID starting with 14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c not found: ID does not exist" containerID="14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"
Oct 30 20:58:53.344220 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.344189    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c"} err="failed to get container status \"14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c\": rpc error: code = NotFound desc = could not find container \"14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c\": container with ID starting with 14d91f62d9cfccfa8582e5ccca50a92e80159ca99b00354fb930584647d3796c not found: ID does not exist"
Oct 30 20:58:53.543720 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.543697    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"]
Oct 30 20:58:53.586273 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:53.586205    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz"]
Oct 30 20:58:55.543506 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:58:55.543475    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" path="/var/lib/kubelet/pods/ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d/volumes"
Oct 30 20:59:01.979608 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:59:01.979570    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:59:11.726149 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:11.726115    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"]
Oct 30 20:59:11.736852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:11.736819    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:59:11.792766 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:11.791879    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f0723-pzgrd"]
Oct 30 20:59:11.863765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:11.852276    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8" podUID="fd69661d-c5d4-446c-ad59-866e2e47adc7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:59:11.884923 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:11.884878    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:59:13.544177 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:13.544142    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab35a934-39fa-44db-b477-d6a7172dc3ea" path="/var/lib/kubelet/pods/ab35a934-39fa-44db-b477-d6a7172dc3ea/volumes"
Oct 30 20:59:17.638505 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:17.638469    5967 generic.go:356] "Generic (PLEG): container finished" podID="0d7dfda0-0021-4b45-bc15-044422fad237" containerID="997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d" exitCode=0
Oct 30 20:59:17.638972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:17.638541    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" event={"ID":"0d7dfda0-0021-4b45-bc15-044422fad237","Type":"ContainerDied","Data":"997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"}
Oct 30 20:59:17.638972 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:17.638938    5967 scope.go:122] "RemoveContainer" containerID="997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"
Oct 30 20:59:17.640484 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:17.640459    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:17.640560 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:17.640524    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:17.640595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:17.640557    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:18.643911 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:18.643877    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"]
Oct 30 20:59:18.644360 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:18.644067    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy" containerID="cri-o://98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59" gracePeriod=30
Oct 30 20:59:18.650268 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:18.650243    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" event={"ID":"0d7dfda0-0021-4b45-bc15-044422fad237","Type":"ContainerStarted","Data":"2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd"}
Oct 30 20:59:19.165093 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.165060    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:59:19.166101 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.166075    5967 patch_prober.go:29] interesting pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 20:59:19.166222 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.166124    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 20:59:19.444992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.444896    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"]
Oct 30 20:59:19.445765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445723    5967 cpu_manager.go:414] "containerMap: removing container" podUID="65d296c8-49bc-4d0c-885f-d781dc5ae26b" containerName="revoke-keys"
Oct 30 20:59:19.445765 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445760    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d296c8-49bc-4d0c-885f-d781dc5ae26b" containerName="revoke-keys"
Oct 30 20:59:19.445952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445775    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy"
Oct 30 20:59:19.445952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445782    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy"
Oct 30 20:59:19.445952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445793    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f4f859df-b5e7-4821-bb2a-7409463fcdcc" containerName="revoke-keys"
Oct 30 20:59:19.445952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445800    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f859df-b5e7-4821-bb2a-7409463fcdcc" containerName="revoke-keys"
Oct 30 20:59:19.445952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445875    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy"
Oct 30 20:59:19.445952 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.445883    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy"
Oct 30 20:59:19.446254 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.446021    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy"
Oct 30 20:59:19.446254 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.446034    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d296c8-49bc-4d0c-885f-d781dc5ae26b" containerName="revoke-keys"
Oct 30 20:59:19.446254 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.446042    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae17a91b-252c-4bce-a0ad-a0e5d2ee9f4d" containerName="istio-proxy"
Oct 30 20:59:19.446254 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.446050    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f859df-b5e7-4821-bb2a-7409463fcdcc" containerName="revoke-keys"
Oct 30 20:59:19.450088 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.450065    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.503031 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.502996    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:59:19.508224 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.507282    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"]
Oct 30 20:59:19.508442 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.508427    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:59:19.523451 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.523421    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9vr\" (UniqueName: \"kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.523715 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.523694    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.523825 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.523770    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.552704 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.552681    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:19.552864 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.552779    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:59:19.624874 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.624848    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9vr\" (UniqueName: \"kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.625109 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.625080    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.625249 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.625132    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.625654 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.625629    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.627420 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.627401    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.696262 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.696199    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9vr\" (UniqueName: \"kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr\") pod \"maas-api-revoke-keys-e2e-api-b-65948c-lgth8\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:19.768242 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:19.768215    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:20.164678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:20.164645    5967 patch_prober.go:29] interesting pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 20:59:20.164884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:20.164721    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 20:59:20.188583 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:20.188539    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"]
Oct 30 20:59:20.200925 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:59:20.200893    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32d5eb8_87b1_453b_a8ea_ecbbddb99a08.slice/crio-5187b0a1ef831232de79ba3f33ad67dbbd0b91ae7200a6d1d9588dceebcd6e1a WatchSource:0}: Error finding container 5187b0a1ef831232de79ba3f33ad67dbbd0b91ae7200a6d1d9588dceebcd6e1a: Status 404 returned error can't find the container with id 5187b0a1ef831232de79ba3f33ad67dbbd0b91ae7200a6d1d9588dceebcd6e1a
Oct 30 20:59:20.666023 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:20.665996    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8" event={"ID":"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08","Type":"ContainerStarted","Data":"4c1e6139a2b3a34be298a4233906678b9b4d09f4d1e1fcf68b866ace2b427fbb"}
Oct 30 20:59:20.666198 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:20.666043    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8" event={"ID":"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08","Type":"ContainerStarted","Data":"5187b0a1ef831232de79ba3f33ad67dbbd0b91ae7200a6d1d9588dceebcd6e1a"}
Oct 30 20:59:20.798354 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:20.798209    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8" podStartSLOduration=1.798188799 podStartE2EDuration="1.798188799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:20.781413755 +0000 UTC m=+2341.382215716" watchObservedRunningTime="2026-10-30 20:59:20.798188799 +0000 UTC m=+2341.398990760"
Oct 30 20:59:21.165225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.165149    5967 patch_prober.go:29] interesting pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 20:59:21.165225 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.165199    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 20:59:21.697475 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.697388    5967 generic.go:356] "Generic (PLEG): container finished" podID="c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" containerID="4c1e6139a2b3a34be298a4233906678b9b4d09f4d1e1fcf68b866ace2b427fbb" exitCode=0
Oct 30 20:59:21.697623 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.697474    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8" event={"ID":"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08","Type":"ContainerDied","Data":"4c1e6139a2b3a34be298a4233906678b9b4d09f4d1e1fcf68b866ace2b427fbb"}
Oct 30 20:59:21.726595 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.726564    5967 scope.go:122] "RemoveContainer" containerID="dea47d12708d6b34305294ed0ccc8d9ca23abd6e54a217d40374e51f01d1f51d"
Oct 30 20:59:21.737029 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.737011    5967 scope.go:122] "RemoveContainer" containerID="f2b844f3300c0178304f4f7127724bede2f66794118d419617e3e15492e77dea"
Oct 30 20:59:21.746436 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.746415    5967 scope.go:122] "RemoveContainer" containerID="7612ff1377d000c213826a036e6d6a17e6b3d0cdbc57abf1bf58b36d9444a17b"
Oct 30 20:59:21.755685 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:21.755668    5967 scope.go:122] "RemoveContainer" containerID="1a54b83d78aa9e1387baa8280e70a716538459fb995ca099284071b8bd25d7b5"
Oct 30 20:59:22.164948 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.164915    5967 patch_prober.go:29] interesting pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 20:59:22.165313 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.164971    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 20:59:22.896995 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.896947    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:22.967167 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.967140    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr\" (UniqueName: \"kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr\") pod \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") "
Oct 30 20:59:22.967325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.967203    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca\") pod \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") "
Oct 30 20:59:22.967325 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.967310    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca\") pod \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\" (UID: \"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08\") "
Oct 30 20:59:22.967573 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.967549    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca" pod "c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" (UID: "c32d5eb8-87b1-453b-a8ea-ecbbddb99a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:22.967775 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.967727    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:22.994986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.994922    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca" pod "c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" (UID: "c32d5eb8-87b1-453b-a8ea-ecbbddb99a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:22.995457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:22.995427    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr" pod "c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" (UID: "c32d5eb8-87b1-453b-a8ea-ecbbddb99a08"). InnerVolumeSpecName "kube-api-access-8r9vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:23.093770 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.068938    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r9vr\" (UniqueName: \"kubernetes.io/projected/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-kube-api-access-8r9vr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:23.093986 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.093865    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:23.164947 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.164914    5967 patch_prober.go:29] interesting pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 20:59:23.165105 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.164964    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 20:59:23.713931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.713899    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8" event={"ID":"c32d5eb8-87b1-453b-a8ea-ecbbddb99a08","Type":"ContainerDied","Data":"5187b0a1ef831232de79ba3f33ad67dbbd0b91ae7200a6d1d9588dceebcd6e1a"}
Oct 30 20:59:23.713931 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.713924    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"
Oct 30 20:59:23.714094 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.713946    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5187b0a1ef831232de79ba3f33ad67dbbd0b91ae7200a6d1d9588dceebcd6e1a"
Oct 30 20:59:23.889609 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.889579    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:59:23.925629 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:23.925610    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:59:24.006617 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006595    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006627    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006707    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006773 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006728    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006782    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006810    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006849    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.006918 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006890    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.007136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.006929    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq\") pod \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\" (UID: \"ce9a7b15-288a-4b84-ac8f-b179c382dc9e\") "
Oct 30 20:59:24.007136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007041    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:24.007136 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007064    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:24.007305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007147    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:24.007305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007231    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:24.007408 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007358    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:24.007904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007666    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.007904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007700    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.007904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007723    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.007904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007767    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.007904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.007783    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.009181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.009163    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:59:24.009319 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.009300    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "kube-api-access-684bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:24.009470 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.009447    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:24.009555 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.009498    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy" pod "ce9a7b15-288a-4b84-ac8f-b179c382dc9e" (UID: "ce9a7b15-288a-4b84-ac8f-b179c382dc9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:24.108641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.108567    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.108641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.108592    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-684bq\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-kube-api-access-684bq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.108641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.108602    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.108641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.108610    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a7b15-288a-4b84-ac8f-b179c382dc9e-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.169837 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.169810    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:59:24.722091 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.722056    5967 generic.go:356] "Generic (PLEG): container finished" podID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerID="98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59" exitCode=0
Oct 30 20:59:24.722295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.722106    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"
Oct 30 20:59:24.722295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.722140    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" event={"ID":"ce9a7b15-288a-4b84-ac8f-b179c382dc9e","Type":"ContainerDied","Data":"98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59"}
Oct 30 20:59:24.722295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.722192    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7" event={"ID":"ce9a7b15-288a-4b84-ac8f-b179c382dc9e","Type":"ContainerDied","Data":"e95bae723129b075f7dcc4e9619251443a69fafa3a935e1e45505dbb17ba72d4"}
Oct 30 20:59:24.722295 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.722213    5967 scope.go:122] "RemoveContainer" containerID="98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59"
Oct 30 20:59:24.723679 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.723658    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 20:59:24.733194 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.733179    5967 scope.go:122] "RemoveContainer" containerID="8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"
Oct 30 20:59:24.742339 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.742323    5967 scope.go:122] "RemoveContainer" containerID="98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59"
Oct 30 20:59:24.742610 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:59:24.742590    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59\": container with ID starting with 98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59 not found: ID does not exist" containerID="98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59"
Oct 30 20:59:24.742666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.742619    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59"} err="failed to get container status \"98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59\": rpc error: code = NotFound desc = could not find container \"98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59\": container with ID starting with 98354efffaf0dd802a5113ce0e685e0b6547c0b68a1a94cdc4bbe9da79701e59 not found: ID does not exist"
Oct 30 20:59:24.742666 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.742634    5967 scope.go:122] "RemoveContainer" containerID="8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"
Oct 30 20:59:24.742894 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:59:24.742876    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94\": container with ID starting with 8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94 not found: ID does not exist" containerID="8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"
Oct 30 20:59:24.742946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.742899    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94"} err="failed to get container status \"8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94\": rpc error: code = NotFound desc = could not find container \"8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94\": container with ID starting with 8df3e4bbe46c6120e365315ef19ea3016eed1ed417f1e320e8b5c9b1c7eadb94 not found: ID does not exist"
Oct 30 20:59:24.885884 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.885847    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"]
Oct 30 20:59:24.935064 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:24.935032    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7"]
Oct 30 20:59:25.543534 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:25.543504    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" path="/var/lib/kubelet/pods/ce9a7b15-288a-4b84-ac8f-b179c382dc9e/volumes"
Oct 30 20:59:28.322946 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:28.322913    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"]
Oct 30 20:59:28.327346 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:28.327318    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a78bf4-drphw"]
Oct 30 20:59:29.543085 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:29.543051    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9d2402-6b56-49de-b2de-8f52f5ced372" path="/var/lib/kubelet/pods/1e9d2402-6b56-49de-b2de-8f52f5ced372/volumes"
Oct 30 20:59:31.980528 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:59:31.980489    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 20:59:34.886522 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:34.886484    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"]
Oct 30 20:59:34.887361 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:34.887325    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" containerID="cri-o://fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd" gracePeriod=30
Oct 30 20:59:36.631386 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631354    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"]
Oct 30 20:59:36.631893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631879    5967 cpu_manager.go:414] "containerMap: removing container" podUID="c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" containerName="revoke-keys"
Oct 30 20:59:36.631893 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631893    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" containerName="revoke-keys"
Oct 30 20:59:36.632034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631908    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy"
Oct 30 20:59:36.632034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631913    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy"
Oct 30 20:59:36.632034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631948    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy"
Oct 30 20:59:36.632034 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.631953    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy"
Oct 30 20:59:36.632153 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.632050    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy"
Oct 30 20:59:36.632153 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.632062    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" containerName="revoke-keys"
Oct 30 20:59:36.636963 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.636945    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683678 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683647    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683800 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683784    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683852 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683817    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683912 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683899    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683949 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683925    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5r52\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683980 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683950    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.683980 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.683966    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.684099 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.684080    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.684132 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.684110    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785463 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785430    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785513    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785538    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5r52\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785570    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785615 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785597    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785666    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785692    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785728    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.785981 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785877    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.786181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785986    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.786181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.785997    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.786181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.786037    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.786181 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.786146    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.786448 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.786427    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.788050 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.788031    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.788289 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.788271    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.837868 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.837840    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"]
Oct 30 20:59:36.842258 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.842232    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5r52\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.842873 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.842845    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pr86v\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:36.949776 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:36.949695    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:37.384641 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.384615    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"]
Oct 30 20:59:37.385794 ip-10-0-130-195 kubenswrapper[5967]: W1030 20:59:37.385732    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9780fc58_0b21_4bd0_81c4_ca1676765f8c.slice/crio-49fe384254d65ec3511d9af80d68447bd933fc36770c291d44b34debe2b2d8a5 WatchSource:0}: Error finding container 49fe384254d65ec3511d9af80d68447bd933fc36770c291d44b34debe2b2d8a5: Status 404 returned error can't find the container with id 49fe384254d65ec3511d9af80d68447bd933fc36770c291d44b34debe2b2d8a5
Oct 30 20:59:37.388341 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.388309    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.388409 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.388377    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.388457 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.388407    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:59:37.634554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.634477    5967 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:59:37.634554 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.634537    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:59:37.818055 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.818007    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" event={"ID":"9780fc58-0b21-4bd0-81c4-ca1676765f8c","Type":"ContainerStarted","Data":"1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e"}
Oct 30 20:59:37.818330 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.818073    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" event={"ID":"9780fc58-0b21-4bd0-81c4-ca1676765f8c","Type":"ContainerStarted","Data":"49fe384254d65ec3511d9af80d68447bd933fc36770c291d44b34debe2b2d8a5"}
Oct 30 20:59:37.945035 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.944892    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podStartSLOduration=1.944866952 podStartE2EDuration="1.944866952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:37.944361304 +0000 UTC m=+2358.545163264" watchObservedRunningTime="2026-10-30 20:59:37.944866952 +0000 UTC m=+2358.545669032"
Oct 30 20:59:37.950260 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:37.950239    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:38.952011 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:38.951978    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pr86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 20:59:38.952472 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:38.952032    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 20:59:39.950866 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:39.950839    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pr86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 20:59:39.951013 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:39.950889    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 20:59:40.202351 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.202324    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:59:40.257871 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.257844    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.257877    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258033 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.257958    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258116    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258168 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258122    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:40.258280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258234    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:40.258280 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258242    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258326    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258380 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258357    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258407    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258436    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l\") pod \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\" (UID: \"48f0e4ca-0fde-4823-a54d-72663bbf30ed\") "
Oct 30 20:59:40.258497 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258446    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:40.258661 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258609    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:40.258729 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258704    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:40.259009 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.258983    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.259145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.259014    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.259145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.259030    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.259145 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.259049    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.259333 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.259152    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.260405 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.260380    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:59:40.260681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.260656    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:40.260681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.260672    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:40.260885 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.260677    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l" pod "48f0e4ca-0fde-4823-a54d-72663bbf30ed" (UID: "48f0e4ca-0fde-4823-a54d-72663bbf30ed"). InnerVolumeSpecName "kube-api-access-9hc5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:40.360850 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.360818    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.361074 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.361055    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hc5l\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-kube-api-access-9hc5l\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.361186 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.361173    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.361283 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.361271    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f0e4ca-0fde-4823-a54d-72663bbf30ed-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.850898 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.850862    5967 generic.go:356] "Generic (PLEG): container finished" podID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerID="fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd" exitCode=0
Oct 30 20:59:40.856904 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.851297    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" event={"ID":"48f0e4ca-0fde-4823-a54d-72663bbf30ed","Type":"ContainerDied","Data":"fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd"}
Oct 30 20:59:40.857153 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.857120    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"
Oct 30 20:59:40.857305 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.857280    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2" event={"ID":"48f0e4ca-0fde-4823-a54d-72663bbf30ed","Type":"ContainerDied","Data":"36bf04ed8ed866199cac3161465cdbba8b5d03b0073a3da3a503b8e6b812ce67"}
Oct 30 20:59:40.857418 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.857406    5967 scope.go:122] "RemoveContainer" containerID="fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd"
Oct 30 20:59:40.916639 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.916268    5967 scope.go:122] "RemoveContainer" containerID="26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"
Oct 30 20:59:40.925992 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.925973    5967 scope.go:122] "RemoveContainer" containerID="fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd"
Oct 30 20:59:40.926262 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:59:40.926240    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd\": container with ID starting with fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd not found: ID does not exist" containerID="fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd"
Oct 30 20:59:40.926335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.926270    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd"} err="failed to get container status \"fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd\": rpc error: code = NotFound desc = could not find container \"fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd\": container with ID starting with fb6f6cc12d23d01329e73179361da7a4a6fdb75d0f3ab2f068b64635ce9575fd not found: ID does not exist"
Oct 30 20:59:40.926335 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.926289    5967 scope.go:122] "RemoveContainer" containerID="26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"
Oct 30 20:59:40.926523 ip-10-0-130-195 kubenswrapper[5967]: E1030 20:59:40.926506    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46\": container with ID starting with 26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46 not found: ID does not exist" containerID="26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"
Oct 30 20:59:40.926581 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.926528    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46"} err="failed to get container status \"26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46\": rpc error: code = NotFound desc = could not find container \"26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46\": container with ID starting with 26f8d57fbb47a90294f1408b761e92d4fc12c93dfe267bfbbf3fccf23399df46 not found: ID does not exist"
Oct 30 20:59:40.950587 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.950563    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pr86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 20:59:40.950681 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:40.950606    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 20:59:41.131546 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:41.131449    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"]
Oct 30 20:59:41.134110 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:41.134087    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2"]
Oct 30 20:59:41.542734 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:41.542695    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" path="/var/lib/kubelet/pods/48f0e4ca-0fde-4823-a54d-72663bbf30ed/volumes"
Oct 30 20:59:41.950427 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:41.950349    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pr86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 20:59:41.950427 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:41.950396    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 20:59:42.950394 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:42.950358    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pr86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 20:59:42.950849 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:42.950414    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 20:59:43.950994 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:43.950959    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pr86v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 20:59:43.951391 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:43.951007    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 20:59:45.628833 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:45.628778    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:45.629410 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:45.629391    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 20:59:45.630250 ip-10-0-130-195 kubenswrapper[5967]: I1030 20:59:45.630229    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 21:00:00.163415 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.163381    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"]
Oct 30 21:00:00.164074 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.164057    5967 cpu_manager.go:414] "containerMap: removing container" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy"
Oct 30 21:00:00.164123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.164076    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy"
Oct 30 21:00:00.164123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.164111    5967 cpu_manager.go:414] "containerMap: removing container" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy"
Oct 30 21:00:00.164123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.164116    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy"
Oct 30 21:00:00.164226 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.164205    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy"
Oct 30 21:00:00.164226 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.164218    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9a7b15-288a-4b84-ac8f-b179c382dc9e" containerName="istio-proxy"
Oct 30 21:00:00.167658 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.167638    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"]
Oct 30 21:00:00.167802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.167773    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.168435 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.168417    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f0e4ca-0fde-4823-a54d-72663bbf30ed" containerName="istio-proxy"
Oct 30 21:00:00.172233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.172203    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"]
Oct 30 21:00:00.172363 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.172274    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.176363 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.176345    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.212511 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.212485    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"]
Oct 30 21:00:00.216676 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.216656    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"]
Oct 30 21:00:00.216809 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.216782    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.220640 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.220620    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"]
Oct 30 21:00:00.220802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.220782    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.220945 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.220923    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bcs\" (UniqueName: \"kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs\") pod \"maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.221066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.220985    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.221066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.221011    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.221066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.221045    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.221233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.221112    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86nz\" (UniqueName: \"kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz\") pod \"maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.221336 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.221321    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfm6\" (UniqueName: \"kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6\") pod \"maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.225232 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.225214    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"]
Oct 30 21:00:00.225318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.225304    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.229825 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.229808    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"]
Oct 30 21:00:00.229936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.229918    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.234027 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.233816    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"]
Oct 30 21:00:00.234027 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.233916    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.237480 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.237453    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.259602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.259577    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:00:00.259731 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.259664    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:00.259903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.259768    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:00.270107 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.270087    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"]
Oct 30 21:00:00.274060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.274041    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"]
Oct 30 21:00:00.276815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.276795    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"]
Oct 30 21:00:00.325727 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.325695    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hsr\" (UniqueName: \"kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr\") pod \"maas-api-key-cleanup-29889900-9gxtk\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.325946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.325760    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bcs\" (UniqueName: \"kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs\") pod \"maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.325946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.325890    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.325946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.325934    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.326123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.325982    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.326123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326027    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.326123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326066    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-9gxtk\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.326283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326123    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86nz\" (UniqueName: \"kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz\") pod \"maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.326283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326210    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.326389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326292    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krxk\" (UniqueName: \"kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk\") pod \"maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.326389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326340    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfm6\" (UniqueName: \"kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6\") pod \"maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.326389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326371    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.326546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326406    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqk8h\" (UniqueName: \"kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h\") pod \"maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.326546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326449    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ppr\" (UniqueName: \"kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr\") pod \"maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.326546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326504    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trwvm\" (UniqueName: \"kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.326705 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326535    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.326705 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.326583    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.327263 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.327129    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndsk\" (UniqueName: \"kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk\") pod \"maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.328271 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.328248    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.328781 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.328762    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.329798 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.329731    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.332373 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.332345    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"]
Oct 30 21:00:00.336240 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.336212    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"]
Oct 30 21:00:00.339549 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.339522    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"]
Oct 30 21:00:00.341848 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.341826    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"]
Oct 30 21:00:00.390145 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.389979    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bcs\" (UniqueName: \"kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs\") pod \"maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.396266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.395496    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86nz\" (UniqueName: \"kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz\") pod \"maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.399113 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.399087    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"]
Oct 30 21:00:00.403441 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.403417    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"]
Oct 30 21:00:00.404281 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.404197    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfm6\" (UniqueName: \"kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6\") pod \"maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.428346 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428284    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.428346 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428335    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krxk\" (UniqueName: \"kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk\") pod \"maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.428539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428370    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.428539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428402    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqk8h\" (UniqueName: \"kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h\") pod \"maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.428539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428435    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ppr\" (UniqueName: \"kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr\") pod \"maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.428726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428640    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trwvm\" (UniqueName: \"kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.428726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428675    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.428726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428715    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.428957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428765    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndsk\" (UniqueName: \"kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk\") pod \"maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.428957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428855    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hsr\" (UniqueName: \"kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr\") pod \"maas-api-key-cleanup-29889900-9gxtk\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.428957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428944    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.429115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.428976    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-9gxtk\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.429178 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.429108    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.429366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.429338    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.429602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.429578    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.429710 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.429690    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-9gxtk\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.429853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.429718    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.430105 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.430087    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.486625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.486410    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:00.486809 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.486784    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krxk\" (UniqueName: \"kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk\") pod \"maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.486904 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.486809    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trwvm\" (UniqueName: \"kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm\") pod \"maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.487488 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.487465    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndsk\" (UniqueName: \"kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk\") pod \"maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.487634 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.487499    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hsr\" (UniqueName: \"kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr\") pod \"maas-api-key-cleanup-29889900-9gxtk\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.487634 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.487554    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqk8h\" (UniqueName: \"kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h\") pod \"maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:00.487849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.487831    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ppr\" (UniqueName: \"kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr\") pod \"maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.495531 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.495514    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:00.505460 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.505434    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:00.531639 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.531606    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:00.543287 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.541520    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:00.552514 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.552486    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:00.573047 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.571184    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:00.576602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.576583    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:00.583970 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:00.583947    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:01.092023 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.091998    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"]
Oct 30 21:00:01.094505 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.094477    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc4a21c_3815_40ad_9220_36d162c5f7f1.slice/crio-f8e6716aeb8b7db8427c9fecd384cc1b8fd288e508aa40b966b541bd124636f5 WatchSource:0}: Error finding container f8e6716aeb8b7db8427c9fecd384cc1b8fd288e508aa40b966b541bd124636f5: Status 404 returned error can't find the container with id f8e6716aeb8b7db8427c9fecd384cc1b8fd288e508aa40b966b541bd124636f5
Oct 30 21:00:01.145818 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.145788    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe91b65f_d75a_4d2a_a526_47ff739cb3ba.slice/crio-4b4f9e9096a73c4927ed7191f476c37ab8dbb1622e0ef710e9dfa4169536d8f2 WatchSource:0}: Error finding container 4b4f9e9096a73c4927ed7191f476c37ab8dbb1622e0ef710e9dfa4169536d8f2: Status 404 returned error can't find the container with id 4b4f9e9096a73c4927ed7191f476c37ab8dbb1622e0ef710e9dfa4169536d8f2
Oct 30 21:00:01.146587 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.146560    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"]
Oct 30 21:00:01.149356 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.149332    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429c2c9_9638_4bd7_8f24_3fd47bf26819.slice/crio-ba1b4adc304d3d93af0133a9f3e0d0fb11b4b982902e4e20381dddab90f1e0fa WatchSource:0}: Error finding container ba1b4adc304d3d93af0133a9f3e0d0fb11b4b982902e4e20381dddab90f1e0fa: Status 404 returned error can't find the container with id ba1b4adc304d3d93af0133a9f3e0d0fb11b4b982902e4e20381dddab90f1e0fa
Oct 30 21:00:01.150613 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.150554    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"]
Oct 30 21:00:01.150816 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.150787    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dddb1b_ba6c_4d29_97a7_3c673bea3dc2.slice/crio-3563e4e6d02944ea508eafa2b55d73b6df0cdd985104a2a1ff748fb75aec72d3 WatchSource:0}: Error finding container 3563e4e6d02944ea508eafa2b55d73b6df0cdd985104a2a1ff748fb75aec72d3: Status 404 returned error can't find the container with id 3563e4e6d02944ea508eafa2b55d73b6df0cdd985104a2a1ff748fb75aec72d3
Oct 30 21:00:01.166524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.166501    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"]
Oct 30 21:00:01.209847 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.196049    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"]
Oct 30 21:00:01.269916 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.269810    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"]
Oct 30 21:00:01.274398 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.274342    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0f4093_51ec_4e34_b3c8_9d0b4e316b27.slice/crio-4de442c15d6a5a1ebe52a330ed24570db0927bf821450804bfda2d5e648735c3 WatchSource:0}: Error finding container 4de442c15d6a5a1ebe52a330ed24570db0927bf821450804bfda2d5e648735c3: Status 404 returned error can't find the container with id 4de442c15d6a5a1ebe52a330ed24570db0927bf821450804bfda2d5e648735c3
Oct 30 21:00:01.294559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.292111    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"]
Oct 30 21:00:01.303097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.303069    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"]
Oct 30 21:00:01.310184 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:01.306874    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"]
Oct 30 21:00:01.313518 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.313493    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c34c9e1_2c16_4e5e_b020_94a13d9fd761.slice/crio-a9b80ac77aa29ee7412507b5321a4afecd2cf1d61e6a120af6bc145fb2bb55c7 WatchSource:0}: Error finding container a9b80ac77aa29ee7412507b5321a4afecd2cf1d61e6a120af6bc145fb2bb55c7: Status 404 returned error can't find the container with id a9b80ac77aa29ee7412507b5321a4afecd2cf1d61e6a120af6bc145fb2bb55c7
Oct 30 21:00:01.372999 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:01.372913    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6a0f13_7f82_4dca_a2be_f53eb5617cad.slice/crio-c50b04588802d99edad8d1e03555acd9986abb9869d50d945dc275883e325859 WatchSource:0}: Error finding container c50b04588802d99edad8d1e03555acd9986abb9869d50d945dc275883e325859: Status 404 returned error can't find the container with id c50b04588802d99edad8d1e03555acd9986abb9869d50d945dc275883e325859
Oct 30 21:00:02.008603 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:02.008535    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:00:02.021585 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.021550    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerStarted","Data":"b87681159d11176165c48fdfad0343c474a8a9035fc72bf5862578af665ea772"}
Oct 30 21:00:02.021807 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.021609    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerStarted","Data":"ba1b4adc304d3d93af0133a9f3e0d0fb11b4b982902e4e20381dddab90f1e0fa"}
Oct 30 21:00:02.023316 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.023293    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerStarted","Data":"50b8b3eac8201cc4848378f325e4158f161d27df927f4492f770a1b8f0ff7b73"}
Oct 30 21:00:02.023434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.023336    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerStarted","Data":"4b4f9e9096a73c4927ed7191f476c37ab8dbb1622e0ef710e9dfa4169536d8f2"}
Oct 30 21:00:02.024591 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.024572    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerStarted","Data":"4de442c15d6a5a1ebe52a330ed24570db0927bf821450804bfda2d5e648735c3"}
Oct 30 21:00:02.026366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.026342    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerStarted","Data":"a9b80ac77aa29ee7412507b5321a4afecd2cf1d61e6a120af6bc145fb2bb55c7"}
Oct 30 21:00:02.028004 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.027977    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerStarted","Data":"12f4d4c89532831af7df0d4ac5bc183cf64f5db632ec177f16bd8a03cbe9da4a"}
Oct 30 21:00:02.028097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.028020    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerStarted","Data":"8154cc04110853d11de7aae759620d50b9d0ce7b0b348b7f8aa10ed8a9fa9615"}
Oct 30 21:00:02.029226 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.029194    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerStarted","Data":"b43700a91313772f4e2915ed052ae919a8612e4de1d0583e0f1461ac0acb0873"}
Oct 30 21:00:02.030831 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.030805    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerStarted","Data":"4f45290d4ac0f914a13cbe82d73c88e4faa7fe1c1440e7bbca20abde450a3222"}
Oct 30 21:00:02.030960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.030850    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerStarted","Data":"f8e6716aeb8b7db8427c9fecd384cc1b8fd288e508aa40b966b541bd124636f5"}
Oct 30 21:00:02.032320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.032300    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerStarted","Data":"c50b04588802d99edad8d1e03555acd9986abb9869d50d945dc275883e325859"}
Oct 30 21:00:02.033907 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.033887    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerStarted","Data":"f170befce92a07e017700cf42872edf84f9a3e9c89ab7d060cc78e59f4b289ac"}
Oct 30 21:00:02.034013 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.033920    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerStarted","Data":"3563e4e6d02944ea508eafa2b55d73b6df0cdd985104a2a1ff748fb75aec72d3"}
Oct 30 21:00:02.135274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.135223    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" podStartSLOduration=2.135211637 podStartE2EDuration="2.135211637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.133318767 +0000 UTC m=+2382.734120727" watchObservedRunningTime="2026-10-30 21:00:02.135211637 +0000 UTC m=+2382.736013595"
Oct 30 21:00:02.246884 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.246799    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" podStartSLOduration=2.246784885 podStartE2EDuration="2.246784885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.24529697 +0000 UTC m=+2382.846098930" watchObservedRunningTime="2026-10-30 21:00:02.246784885 +0000 UTC m=+2382.847586845"
Oct 30 21:00:02.361667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.361621    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" podStartSLOduration=2.36160822 podStartE2EDuration="2.36160822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.35805571 +0000 UTC m=+2382.958857669" watchObservedRunningTime="2026-10-30 21:00:02.36160822 +0000 UTC m=+2382.962410179"
Oct 30 21:00:02.475301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.475249    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" podStartSLOduration=2.475236634 podStartE2EDuration="2.475236634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.471697592 +0000 UTC m=+2383.072499552" watchObservedRunningTime="2026-10-30 21:00:02.475236634 +0000 UTC m=+2383.076038593"
Oct 30 21:00:02.585785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:02.585721    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" podStartSLOduration=2.58570688 podStartE2EDuration="2.58570688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.583855817 +0000 UTC m=+2383.184657775" watchObservedRunningTime="2026-10-30 21:00:02.58570688 +0000 UTC m=+2383.186508839"
Oct 30 21:00:03.042946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.042914    5967 generic.go:356] "Generic (PLEG): container finished" podID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerID="b87681159d11176165c48fdfad0343c474a8a9035fc72bf5862578af665ea772" exitCode=60
Oct 30 21:00:03.043145 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.042999    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerDied","Data":"b87681159d11176165c48fdfad0343c474a8a9035fc72bf5862578af665ea772"}
Oct 30 21:00:03.043258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.043241    5967 scope.go:122] "RemoveContainer" containerID="b87681159d11176165c48fdfad0343c474a8a9035fc72bf5862578af665ea772"
Oct 30 21:00:03.044957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.044809    5967 generic.go:356] "Generic (PLEG): container finished" podID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerID="50b8b3eac8201cc4848378f325e4158f161d27df927f4492f770a1b8f0ff7b73" exitCode=60
Oct 30 21:00:03.044957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.044894    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerDied","Data":"50b8b3eac8201cc4848378f325e4158f161d27df927f4492f770a1b8f0ff7b73"}
Oct 30 21:00:03.045404 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.045386    5967 scope.go:122] "RemoveContainer" containerID="50b8b3eac8201cc4848378f325e4158f161d27df927f4492f770a1b8f0ff7b73"
Oct 30 21:00:03.047101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.047039    5967 generic.go:356] "Generic (PLEG): container finished" podID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerID="ea54b02acbcac32193c87d3f022b77fb5da8c057daf2a28ed8bad72df8f56cd9" exitCode=60
Oct 30 21:00:03.047245 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.047204    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerDied","Data":"ea54b02acbcac32193c87d3f022b77fb5da8c057daf2a28ed8bad72df8f56cd9"}
Oct 30 21:00:03.047371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.047361    5967 scope.go:122] "RemoveContainer" containerID="ea54b02acbcac32193c87d3f022b77fb5da8c057daf2a28ed8bad72df8f56cd9"
Oct 30 21:00:03.048997 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.048978    5967 generic.go:356] "Generic (PLEG): container finished" podID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerID="398f02a5e5b9744e74b0229b348d114d37835809c6e65474cb3d11f610d084db" exitCode=60
Oct 30 21:00:03.049077 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.049056    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerDied","Data":"398f02a5e5b9744e74b0229b348d114d37835809c6e65474cb3d11f610d084db"}
Oct 30 21:00:03.049220 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.049208    5967 scope.go:122] "RemoveContainer" containerID="398f02a5e5b9744e74b0229b348d114d37835809c6e65474cb3d11f610d084db"
Oct 30 21:00:03.050638 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.050622    5967 generic.go:356] "Generic (PLEG): container finished" podID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerID="12f4d4c89532831af7df0d4ac5bc183cf64f5db632ec177f16bd8a03cbe9da4a" exitCode=60
Oct 30 21:00:03.050733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.050703    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerDied","Data":"12f4d4c89532831af7df0d4ac5bc183cf64f5db632ec177f16bd8a03cbe9da4a"}
Oct 30 21:00:03.050886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.050871    5967 scope.go:122] "RemoveContainer" containerID="12f4d4c89532831af7df0d4ac5bc183cf64f5db632ec177f16bd8a03cbe9da4a"
Oct 30 21:00:03.052382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.052357    5967 generic.go:356] "Generic (PLEG): container finished" podID="631243a2-00e9-44e7-acb1-28f36973d765" containerID="effd6878738f9f1b09b878add6349fb8ea4fa6fd57c642ca6002eb52a771abba" exitCode=60
Oct 30 21:00:03.052474 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.052430    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerDied","Data":"effd6878738f9f1b09b878add6349fb8ea4fa6fd57c642ca6002eb52a771abba"}
Oct 30 21:00:03.052628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.052614    5967 scope.go:122] "RemoveContainer" containerID="effd6878738f9f1b09b878add6349fb8ea4fa6fd57c642ca6002eb52a771abba"
Oct 30 21:00:03.054011 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.053989    5967 generic.go:356] "Generic (PLEG): container finished" podID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerID="4f45290d4ac0f914a13cbe82d73c88e4faa7fe1c1440e7bbca20abde450a3222" exitCode=60
Oct 30 21:00:03.054293 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.054276    5967 scope.go:122] "RemoveContainer" containerID="4f45290d4ac0f914a13cbe82d73c88e4faa7fe1c1440e7bbca20abde450a3222"
Oct 30 21:00:03.054521 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.054494    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerDied","Data":"4f45290d4ac0f914a13cbe82d73c88e4faa7fe1c1440e7bbca20abde450a3222"}
Oct 30 21:00:03.056171 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.056152    5967 generic.go:356] "Generic (PLEG): container finished" podID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerID="482f8bcaac3c664fbf7b54f6680025feca0c295752615d66d2a40dbd7531026d" exitCode=60
Oct 30 21:00:03.056265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.056174    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerDied","Data":"482f8bcaac3c664fbf7b54f6680025feca0c295752615d66d2a40dbd7531026d"}
Oct 30 21:00:03.056425 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.056409    5967 scope.go:122] "RemoveContainer" containerID="482f8bcaac3c664fbf7b54f6680025feca0c295752615d66d2a40dbd7531026d"
Oct 30 21:00:03.057876 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.057857    5967 generic.go:356] "Generic (PLEG): container finished" podID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerID="f170befce92a07e017700cf42872edf84f9a3e9c89ab7d060cc78e59f4b289ac" exitCode=60
Oct 30 21:00:03.057960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.057932    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerDied","Data":"f170befce92a07e017700cf42872edf84f9a3e9c89ab7d060cc78e59f4b289ac"}
Oct 30 21:00:03.058127 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:03.058113    5967 scope.go:122] "RemoveContainer" containerID="f170befce92a07e017700cf42872edf84f9a3e9c89ab7d060cc78e59f4b289ac"
Oct 30 21:00:04.067353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.067319    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerStarted","Data":"e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8"}
Oct 30 21:00:04.075549 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.075516    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerStarted","Data":"1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5"}
Oct 30 21:00:04.078061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.078030    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerStarted","Data":"3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8"}
Oct 30 21:00:04.080239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.080214    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerStarted","Data":"94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e"}
Oct 30 21:00:04.082260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.082234    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerStarted","Data":"8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0"}
Oct 30 21:00:04.084165 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.084142    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerStarted","Data":"38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1"}
Oct 30 21:00:04.085950 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.085928    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerStarted","Data":"cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855"}
Oct 30 21:00:04.087787 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.087766    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerStarted","Data":"20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797"}
Oct 30 21:00:04.089542 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.089522    5967 generic.go:356] "Generic (PLEG): container finished" podID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerID="a7878ee1e6b204c4555b63b85039924b19f4858a7f0e1fd5358990e371392bc3" exitCode=60
Oct 30 21:00:04.089625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.089563    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerDied","Data":"a7878ee1e6b204c4555b63b85039924b19f4858a7f0e1fd5358990e371392bc3"}
Oct 30 21:00:04.089625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.089606    5967 scope.go:122] "RemoveContainer" containerID="12f4d4c89532831af7df0d4ac5bc183cf64f5db632ec177f16bd8a03cbe9da4a"
Oct 30 21:00:04.089838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.089822    5967 scope.go:122] "RemoveContainer" containerID="a7878ee1e6b204c4555b63b85039924b19f4858a7f0e1fd5358990e371392bc3"
Oct 30 21:00:04.090054 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:04.090037    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94_odh-ai-gateway-infra(05a5e5ce-51e3-40a3-bebe-ed036d68704e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e"
Oct 30 21:00:04.340486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.340438    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" podStartSLOduration=4.340423137 podStartE2EDuration="4.340423137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:04.337954802 +0000 UTC m=+2384.938756762" watchObservedRunningTime="2026-10-30 21:00:04.340423137 +0000 UTC m=+2384.941225134"
Oct 30 21:00:04.461891 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.461839    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" podStartSLOduration=4.461820897 podStartE2EDuration="4.461820897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:04.458763453 +0000 UTC m=+2385.059565408" watchObservedRunningTime="2026-10-30 21:00:04.461820897 +0000 UTC m=+2385.062622859"
Oct 30 21:00:04.928846 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:04.928728    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" podStartSLOduration=4.928714867 podStartE2EDuration="4.928714867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:04.927520595 +0000 UTC m=+2385.528322558" watchObservedRunningTime="2026-10-30 21:00:04.928714867 +0000 UTC m=+2385.529516805"
Oct 30 21:00:05.098478 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.098447    5967 generic.go:356] "Generic (PLEG): container finished" podID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerID="8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0" exitCode=60
Oct 30 21:00:05.098968 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.098493    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerDied","Data":"8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0"}
Oct 30 21:00:05.098968 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.098549    5967 scope.go:122] "RemoveContainer" containerID="b87681159d11176165c48fdfad0343c474a8a9035fc72bf5862578af665ea772"
Oct 30 21:00:05.098968 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.098881    5967 scope.go:122] "RemoveContainer" containerID="8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0"
Oct 30 21:00:05.099238 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.099216    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv_odh-ai-gateway-infra(5429c2c9-9638-4bd7-8f24-3fd47bf26819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819"
Oct 30 21:00:05.100641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.100623    5967 generic.go:356] "Generic (PLEG): container finished" podID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerID="38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1" exitCode=60
Oct 30 21:00:05.100790 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.100716    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerDied","Data":"38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1"}
Oct 30 21:00:05.100974 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.100960    5967 scope.go:122] "RemoveContainer" containerID="38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1"
Oct 30 21:00:05.101324 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.101300    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4_odh-ai-gateway-infra(fe91b65f-d75a-4d2a-a526-47ff739cb3ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba"
Oct 30 21:00:05.102505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.102486    5967 generic.go:356] "Generic (PLEG): container finished" podID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerID="cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855" exitCode=60
Oct 30 21:00:05.102592 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.102573    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerDied","Data":"cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855"}
Oct 30 21:00:05.102806 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.102793    5967 scope.go:122] "RemoveContainer" containerID="cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855"
Oct 30 21:00:05.103100 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.103080    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-9gxtk_odh-ai-gateway-infra(ac0f4093-51ec-4e34-b3c8-9d0b4e316b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27"
Oct 30 21:00:05.104632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.104614    5967 generic.go:356] "Generic (PLEG): container finished" podID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerID="20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797" exitCode=60
Oct 30 21:00:05.104706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.104674    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerDied","Data":"20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797"}
Oct 30 21:00:05.104893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.104878    5967 scope.go:122] "RemoveContainer" containerID="20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797"
Oct 30 21:00:05.105126 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.105110    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m_odh-ai-gateway-infra(2c34c9e1-2c16-4e5e-b020-94a13d9fd761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761"
Oct 30 21:00:05.106575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.106556    5967 scope.go:122] "RemoveContainer" containerID="a7878ee1e6b204c4555b63b85039924b19f4858a7f0e1fd5358990e371392bc3"
Oct 30 21:00:05.106822 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.106802    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94_odh-ai-gateway-infra(05a5e5ce-51e3-40a3-bebe-ed036d68704e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e"
Oct 30 21:00:05.108313 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.108295    5967 generic.go:356] "Generic (PLEG): container finished" podID="631243a2-00e9-44e7-acb1-28f36973d765" containerID="e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8" exitCode=60
Oct 30 21:00:05.108399 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.108370    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerDied","Data":"e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8"}
Oct 30 21:00:05.108569 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.108539    5967 scope.go:122] "RemoveContainer" containerID="e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8"
Oct 30 21:00:05.108805 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.108783    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds_odh-ai-gateway-infra(631243a2-00e9-44e7-acb1-28f36973d765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" podUID="631243a2-00e9-44e7-acb1-28f36973d765"
Oct 30 21:00:05.110438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.110415    5967 generic.go:356] "Generic (PLEG): container finished" podID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerID="1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5" exitCode=60
Oct 30 21:00:05.110517 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.110502    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerDied","Data":"1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5"}
Oct 30 21:00:05.110687 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.110671    5967 scope.go:122] "RemoveContainer" containerID="1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5"
Oct 30 21:00:05.110911 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.110894    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f_odh-ai-gateway-infra(7bc4a21c-3815-40ad-9220-36d162c5f7f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1"
Oct 30 21:00:05.112773 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.112726    5967 generic.go:356] "Generic (PLEG): container finished" podID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerID="3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8" exitCode=60
Oct 30 21:00:05.112873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.112787    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerDied","Data":"3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8"}
Oct 30 21:00:05.112997 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.112983    5967 scope.go:122] "RemoveContainer" containerID="3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8"
Oct 30 21:00:05.113195 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.113175    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk_odh-ai-gateway-infra(cc6a0f13-7f82-4dca-a2be-f53eb5617cad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad"
Oct 30 21:00:05.113246 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.113197    5967 scope.go:122] "RemoveContainer" containerID="50b8b3eac8201cc4848378f325e4158f161d27df927f4492f770a1b8f0ff7b73"
Oct 30 21:00:05.114999 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.114982    5967 generic.go:356] "Generic (PLEG): container finished" podID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerID="94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e" exitCode=60
Oct 30 21:00:05.115081 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.115009    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerDied","Data":"94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e"}
Oct 30 21:00:05.115202 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.115190    5967 scope.go:122] "RemoveContainer" containerID="94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e"
Oct 30 21:00:05.115365 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:05.115350    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw_odh-ai-gateway-infra(45dddb1b-ba6c-4d29-97a7-3c673bea3dc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2"
Oct 30 21:00:05.126234 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.126210    5967 scope.go:122] "RemoveContainer" containerID="ea54b02acbcac32193c87d3f022b77fb5da8c057daf2a28ed8bad72df8f56cd9"
Oct 30 21:00:05.139285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.139268    5967 scope.go:122] "RemoveContainer" containerID="398f02a5e5b9744e74b0229b348d114d37835809c6e65474cb3d11f610d084db"
Oct 30 21:00:05.151533 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.151515    5967 scope.go:122] "RemoveContainer" containerID="effd6878738f9f1b09b878add6349fb8ea4fa6fd57c642ca6002eb52a771abba"
Oct 30 21:00:05.161081 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.161035    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" podStartSLOduration=5.161020631 podStartE2EDuration="5.161020631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:05.159793102 +0000 UTC m=+2385.760595063" watchObservedRunningTime="2026-10-30 21:00:05.161020631 +0000 UTC m=+2385.761822592"
Oct 30 21:00:05.164468 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.164454    5967 scope.go:122] "RemoveContainer" containerID="4f45290d4ac0f914a13cbe82d73c88e4faa7fe1c1440e7bbca20abde450a3222"
Oct 30 21:00:05.177486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.177461    5967 scope.go:122] "RemoveContainer" containerID="482f8bcaac3c664fbf7b54f6680025feca0c295752615d66d2a40dbd7531026d"
Oct 30 21:00:05.190646 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:05.190620    5967 scope.go:122] "RemoveContainer" containerID="f170befce92a07e017700cf42872edf84f9a3e9c89ab7d060cc78e59f4b289ac"
Oct 30 21:00:06.123766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.123703    5967 scope.go:122] "RemoveContainer" containerID="cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855"
Oct 30 21:00:06.124182 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.124039    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-9gxtk_odh-ai-gateway-infra(ac0f4093-51ec-4e34-b3c8-9d0b4e316b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27"
Oct 30 21:00:06.125489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.125468    5967 scope.go:122] "RemoveContainer" containerID="20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797"
Oct 30 21:00:06.125732 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.125711    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m_odh-ai-gateway-infra(2c34c9e1-2c16-4e5e-b020-94a13d9fd761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761"
Oct 30 21:00:06.127042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.127022    5967 scope.go:122] "RemoveContainer" containerID="e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8"
Oct 30 21:00:06.127234 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.127209    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds_odh-ai-gateway-infra(631243a2-00e9-44e7-acb1-28f36973d765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" podUID="631243a2-00e9-44e7-acb1-28f36973d765"
Oct 30 21:00:06.128728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.128700    5967 scope.go:122] "RemoveContainer" containerID="1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5"
Oct 30 21:00:06.128955 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.128932    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f_odh-ai-gateway-infra(7bc4a21c-3815-40ad-9220-36d162c5f7f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1"
Oct 30 21:00:06.130348 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.130332    5967 scope.go:122] "RemoveContainer" containerID="3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8"
Oct 30 21:00:06.130540 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.130518    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk_odh-ai-gateway-infra(cc6a0f13-7f82-4dca-a2be-f53eb5617cad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad"
Oct 30 21:00:06.131816 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.131798    5967 scope.go:122] "RemoveContainer" containerID="94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e"
Oct 30 21:00:06.132008 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.131990    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw_odh-ai-gateway-infra(45dddb1b-ba6c-4d29-97a7-3c673bea3dc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2"
Oct 30 21:00:06.133328 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.133311    5967 scope.go:122] "RemoveContainer" containerID="8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0"
Oct 30 21:00:06.133522 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.133501    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv_odh-ai-gateway-infra(5429c2c9-9638-4bd7-8f24-3fd47bf26819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819"
Oct 30 21:00:06.134961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.134945    5967 scope.go:122] "RemoveContainer" containerID="38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1"
Oct 30 21:00:06.135164 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:06.135133    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4_odh-ai-gateway-infra(fe91b65f-d75a-4d2a-a526-47ff739cb3ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba"
Oct 30 21:00:06.425109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.425032    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"]
Oct 30 21:00:06.430485 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.430463    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.482785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.482722    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:06.485400 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.485376    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"]
Oct 30 21:00:06.524345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.524317    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.524499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.524469    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.524540 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.524509    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbcw7\" (UniqueName: \"kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.625516 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.625489    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.625672 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.625656    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.625725 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.625683    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbcw7\" (UniqueName: \"kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.626246 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.626217    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.628114 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.628096    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.682305 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.682243    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbcw7\" (UniqueName: \"kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7\") pod \"maas-api-revoke-sub-478433c7-n7tg9\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:06.743841 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:06.743819    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:07.117452 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:07.117427    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"]
Oct 30 21:00:07.118812 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:07.118780    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9564c9_806b_4014_aeab_a5904949e987.slice/crio-541ac2d3236497db2ac445f7d916ffa1a23ccf6dd95879bbf47a7467ebb2f60d WatchSource:0}: Error finding container 541ac2d3236497db2ac445f7d916ffa1a23ccf6dd95879bbf47a7467ebb2f60d: Status 404 returned error can't find the container with id 541ac2d3236497db2ac445f7d916ffa1a23ccf6dd95879bbf47a7467ebb2f60d
Oct 30 21:00:07.143489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:07.143463    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9" event={"ID":"5b9564c9-806b-4014-aeab-a5904949e987","Type":"ContainerStarted","Data":"541ac2d3236497db2ac445f7d916ffa1a23ccf6dd95879bbf47a7467ebb2f60d"}
Oct 30 21:00:08.152936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:08.152906    5967 generic.go:356] "Generic (PLEG): container finished" podID="5b9564c9-806b-4014-aeab-a5904949e987" containerID="f61280a4dcb4ed2c75f0a7e26131aa23a3aaac46f45e470781fe687e205e25d5" exitCode=0
Oct 30 21:00:08.153277 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:08.152958    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9" event={"ID":"5b9564c9-806b-4014-aeab-a5904949e987","Type":"ContainerDied","Data":"f61280a4dcb4ed2c75f0a7e26131aa23a3aaac46f45e470781fe687e205e25d5"}
Oct 30 21:00:09.305767 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.305724    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:09.460103 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.460034    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca\") pod \"5b9564c9-806b-4014-aeab-a5904949e987\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") "
Oct 30 21:00:09.460103 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.460091    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7\" (UniqueName: \"kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7\") pod \"5b9564c9-806b-4014-aeab-a5904949e987\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") "
Oct 30 21:00:09.460311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.460166    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca\") pod \"5b9564c9-806b-4014-aeab-a5904949e987\" (UID: \"5b9564c9-806b-4014-aeab-a5904949e987\") "
Oct 30 21:00:09.460584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.460552    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca" pod "5b9564c9-806b-4014-aeab-a5904949e987" (UID: "5b9564c9-806b-4014-aeab-a5904949e987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:09.462209 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.462187    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca" pod "5b9564c9-806b-4014-aeab-a5904949e987" (UID: "5b9564c9-806b-4014-aeab-a5904949e987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:09.462693 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.462678    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7" pod "5b9564c9-806b-4014-aeab-a5904949e987" (UID: "5b9564c9-806b-4014-aeab-a5904949e987"). InnerVolumeSpecName "kube-api-access-mbcw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:09.561264 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.561239    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:09.561361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.561265    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbcw7\" (UniqueName: \"kubernetes.io/projected/5b9564c9-806b-4014-aeab-a5904949e987-kube-api-access-mbcw7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:09.561361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:09.561279    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9564c9-806b-4014-aeab-a5904949e987-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:10.167594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:10.167555    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9" event={"ID":"5b9564c9-806b-4014-aeab-a5904949e987","Type":"ContainerDied","Data":"541ac2d3236497db2ac445f7d916ffa1a23ccf6dd95879bbf47a7467ebb2f60d"}
Oct 30 21:00:10.167594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:10.167600    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541ac2d3236497db2ac445f7d916ffa1a23ccf6dd95879bbf47a7467ebb2f60d"
Oct 30 21:00:10.167833 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:10.167562    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"
Oct 30 21:00:15.535960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:15.535920    5967 scope.go:122] "RemoveContainer" containerID="38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1"
Oct 30 21:00:15.536448 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:15.536036    5967 scope.go:122] "RemoveContainer" containerID="94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e"
Oct 30 21:00:16.212659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:16.212548    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerStarted","Data":"11302bd84699b955f4ff5f5c143b27fbd884f741d5f5ad203f8fe958ee20fe4a"}
Oct 30 21:00:16.214953 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:16.214923    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerStarted","Data":"91c8faac1f9acd95bc2fb7502d6c447a4321ed7293e9027aca3ed4457e03c4b2"}
Oct 30 21:00:16.536208 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:16.536184    5967 scope.go:122] "RemoveContainer" containerID="e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8"
Oct 30 21:00:16.536596 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:16.536308    5967 scope.go:122] "RemoveContainer" containerID="1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5"
Oct 30 21:00:17.223331 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.223295    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerStarted","Data":"bf239fed917ec216a8ebcbb5f8737541420f077b75ebad3c969b94324351b9bf"}
Oct 30 21:00:17.225185 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.225161    5967 generic.go:356] "Generic (PLEG): container finished" podID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerID="11302bd84699b955f4ff5f5c143b27fbd884f741d5f5ad203f8fe958ee20fe4a" exitCode=60
Oct 30 21:00:17.225305 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.225235    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerDied","Data":"11302bd84699b955f4ff5f5c143b27fbd884f741d5f5ad203f8fe958ee20fe4a"}
Oct 30 21:00:17.225305 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.225286    5967 scope.go:122] "RemoveContainer" containerID="94f5a0b49fdf996e335cf8efd0801fcebdc007fc3774f5ea02c032ff3ae2245e"
Oct 30 21:00:17.225584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.225560    5967 scope.go:122] "RemoveContainer" containerID="11302bd84699b955f4ff5f5c143b27fbd884f741d5f5ad203f8fe958ee20fe4a"
Oct 30 21:00:17.225941 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:17.225887    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw_odh-ai-gateway-infra(45dddb1b-ba6c-4d29-97a7-3c673bea3dc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2"
Oct 30 21:00:17.227369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.227351    5967 generic.go:356] "Generic (PLEG): container finished" podID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerID="91c8faac1f9acd95bc2fb7502d6c447a4321ed7293e9027aca3ed4457e03c4b2" exitCode=60
Oct 30 21:00:17.227479 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.227430    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerDied","Data":"91c8faac1f9acd95bc2fb7502d6c447a4321ed7293e9027aca3ed4457e03c4b2"}
Oct 30 21:00:17.227753 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.227723    5967 scope.go:122] "RemoveContainer" containerID="91c8faac1f9acd95bc2fb7502d6c447a4321ed7293e9027aca3ed4457e03c4b2"
Oct 30 21:00:17.228044 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:17.228026    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4_odh-ai-gateway-infra(fe91b65f-d75a-4d2a-a526-47ff739cb3ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba"
Oct 30 21:00:17.229259 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.229236    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerStarted","Data":"c0e27fc713d904a0ff68bd992b9f1ba92333ce6f429ade6cbb9ee7927e608e66"}
Oct 30 21:00:17.238453 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.238438    5967 scope.go:122] "RemoveContainer" containerID="38f36ff9f58f756215080ae715b8570580fb12106775d8cd5a4522be412811f1"
Oct 30 21:00:17.338704 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.338681    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"]
Oct 30 21:00:17.456072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.456040    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"]
Oct 30 21:00:17.535546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.535520    5967 scope.go:122] "RemoveContainer" containerID="cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855"
Oct 30 21:00:17.535691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.535621    5967 scope.go:122] "RemoveContainer" containerID="8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0"
Oct 30 21:00:17.535817 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.535800    5967 scope.go:122] "RemoveContainer" containerID="20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797"
Oct 30 21:00:17.709038 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.709002    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"]
Oct 30 21:00:17.710007 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.709986    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5b9564c9-806b-4014-aeab-a5904949e987" containerName="revoke-keys"
Oct 30 21:00:17.710070 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.710010    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9564c9-806b-4014-aeab-a5904949e987" containerName="revoke-keys"
Oct 30 21:00:17.710234 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.710222    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9564c9-806b-4014-aeab-a5904949e987" containerName="revoke-keys"
Oct 30 21:00:17.714385 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.714367    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.756134 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.756083    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.756312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.756292    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdcv8\" (UniqueName: \"kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.756408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.756393    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.765846 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.765823    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:17.768056 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.768031    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"]
Oct 30 21:00:17.857176 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.857147    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdcv8\" (UniqueName: \"kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.857327 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.857211    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.857327 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.857249    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.857885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.857867    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.859613 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.859587    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:17.969616 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:17.969590    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdcv8\" (UniqueName: \"kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:18.198700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.198668    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:18.239710 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.239682    5967 generic.go:356] "Generic (PLEG): container finished" podID="631243a2-00e9-44e7-acb1-28f36973d765" containerID="c0e27fc713d904a0ff68bd992b9f1ba92333ce6f429ade6cbb9ee7927e608e66" exitCode=60
Oct 30 21:00:18.239903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.239766    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerDied","Data":"c0e27fc713d904a0ff68bd992b9f1ba92333ce6f429ade6cbb9ee7927e608e66"}
Oct 30 21:00:18.239903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.239823    5967 scope.go:122] "RemoveContainer" containerID="e71d6e67b5f5fbf9b15ade79dcbb863dfcb45483a9f434ca0466d55f7c64b3a8"
Oct 30 21:00:18.240267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.240248    5967 scope.go:122] "RemoveContainer" containerID="c0e27fc713d904a0ff68bd992b9f1ba92333ce6f429ade6cbb9ee7927e608e66"
Oct 30 21:00:18.240646 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:18.240612    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds_odh-ai-gateway-infra(631243a2-00e9-44e7-acb1-28f36973d765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" podUID="631243a2-00e9-44e7-acb1-28f36973d765"
Oct 30 21:00:18.243159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.242191    5967 generic.go:356] "Generic (PLEG): container finished" podID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerID="bf239fed917ec216a8ebcbb5f8737541420f077b75ebad3c969b94324351b9bf" exitCode=60
Oct 30 21:00:18.243159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.242300    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerDied","Data":"bf239fed917ec216a8ebcbb5f8737541420f077b75ebad3c969b94324351b9bf"}
Oct 30 21:00:18.243159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.242670    5967 scope.go:122] "RemoveContainer" containerID="bf239fed917ec216a8ebcbb5f8737541420f077b75ebad3c969b94324351b9bf"
Oct 30 21:00:18.243159 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:18.243005    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f_odh-ai-gateway-infra(7bc4a21c-3815-40ad-9220-36d162c5f7f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1"
Oct 30 21:00:18.246870 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.246846    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerStarted","Data":"4dab0cd7d5106acf64c7868069cd1dcc27289055badfe6ff528f74ac930e4d4c"}
Oct 30 21:00:18.250723 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.250703    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerStarted","Data":"ad3b32aa29773095ff1c0becc21f0a5ed487a6eb6f0b58042166c0958b2febdb"}
Oct 30 21:00:18.254838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.253337    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerStarted","Data":"2f3e547384ef0956f91fb7efd8faade4f613130d1495408a12b2aa7c3e079c3b"}
Oct 30 21:00:18.263186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.262050    5967 scope.go:122] "RemoveContainer" containerID="1a768e9f9c666dd01f17fb727c6b71a48dc587906f22c8fcd6609377e4f166c5"
Oct 30 21:00:18.354126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.354094    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"]
Oct 30 21:00:18.420603 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.420575    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:18.458266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.458240    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:18.488368 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.488345    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca\") pod \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") "
Oct 30 21:00:18.488534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.488494    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz\" (UniqueName: \"kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz\") pod \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") "
Oct 30 21:00:18.488534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.488530    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h\" (UniqueName: \"kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h\") pod \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\" (UID: \"fe91b65f-d75a-4d2a-a526-47ff739cb3ba\") "
Oct 30 21:00:18.488664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.488617    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca\") pod \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\" (UID: \"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2\") "
Oct 30 21:00:18.488791 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.488768    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca" pod "fe91b65f-d75a-4d2a-a526-47ff739cb3ba" (UID: "fe91b65f-d75a-4d2a-a526-47ff739cb3ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:18.489031 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.489011    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca" pod "45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" (UID: "45dddb1b-ba6c-4d29-97a7-3c673bea3dc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:18.489194 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.489147    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.489194 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.489164    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.490732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.490713    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz" pod "45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" (UID: "45dddb1b-ba6c-4d29-97a7-3c673bea3dc2"). InnerVolumeSpecName "kube-api-access-h86nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:18.490840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.490754    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h" pod "fe91b65f-d75a-4d2a-a526-47ff739cb3ba" (UID: "fe91b65f-d75a-4d2a-a526-47ff739cb3ba"). InnerVolumeSpecName "kube-api-access-mqk8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:18.536186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.536165    5967 scope.go:122] "RemoveContainer" containerID="a7878ee1e6b204c4555b63b85039924b19f4858a7f0e1fd5358990e371392bc3"
Oct 30 21:00:18.536343 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.536323    5967 scope.go:122] "RemoveContainer" containerID="3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8"
Oct 30 21:00:18.580585 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.580556    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"]
Oct 30 21:00:18.590902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.590694    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86nz\" (UniqueName: \"kubernetes.io/projected/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2-kube-api-access-h86nz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.590902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.590719    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqk8h\" (UniqueName: \"kubernetes.io/projected/fe91b65f-d75a-4d2a-a526-47ff739cb3ba-kube-api-access-mqk8h\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.595815 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:18.595789    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2cadff_e237_4e97_af50_839f84aa4dee.slice/crio-54c8a0f0bf581e08e2c9c36072039b97f55c7760f81c6b9a39c66df4b5b15a87 WatchSource:0}: Error finding container 54c8a0f0bf581e08e2c9c36072039b97f55c7760f81c6b9a39c66df4b5b15a87: Status 404 returned error can't find the container with id 54c8a0f0bf581e08e2c9c36072039b97f55c7760f81c6b9a39c66df4b5b15a87
Oct 30 21:00:18.691454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:18.691423    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"]
Oct 30 21:00:19.262317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.262287    5967 generic.go:356] "Generic (PLEG): container finished" podID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerID="2f3e547384ef0956f91fb7efd8faade4f613130d1495408a12b2aa7c3e079c3b" exitCode=60
Oct 30 21:00:19.262758 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.262360    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerDied","Data":"2f3e547384ef0956f91fb7efd8faade4f613130d1495408a12b2aa7c3e079c3b"}
Oct 30 21:00:19.262758 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.262426    5967 scope.go:122] "RemoveContainer" containerID="20d2c358fcb6bd50d9a365850a8ff15625999a6138ff3a0c202b9ae5951fc797"
Oct 30 21:00:19.262889 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.262831    5967 scope.go:122] "RemoveContainer" containerID="2f3e547384ef0956f91fb7efd8faade4f613130d1495408a12b2aa7c3e079c3b"
Oct 30 21:00:19.263204 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:19.263175    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m_odh-ai-gateway-infra(2c34c9e1-2c16-4e5e-b020-94a13d9fd761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761"
Oct 30 21:00:19.264448 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.264428    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerStarted","Data":"2fe10b681d374e532a5072d64feda88442828a22bc5098e3c55f2d794ec25b9d"}
Oct 30 21:00:19.270057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.270036    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerStarted","Data":"d5c31b3074dee2faa7f27da30b343fc7342b89e185a5f2e336f03b865b788aa3"}
Oct 30 21:00:19.272074 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.272001    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"
Oct 30 21:00:19.272444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.272352    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw" event={"ID":"45dddb1b-ba6c-4d29-97a7-3c673bea3dc2","Type":"ContainerDied","Data":"3563e4e6d02944ea508eafa2b55d73b6df0cdd985104a2a1ff748fb75aec72d3"}
Oct 30 21:00:19.277861 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.277831    5967 generic.go:356] "Generic (PLEG): container finished" podID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerID="4dab0cd7d5106acf64c7868069cd1dcc27289055badfe6ff528f74ac930e4d4c" exitCode=60
Oct 30 21:00:19.278267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.277966    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerDied","Data":"4dab0cd7d5106acf64c7868069cd1dcc27289055badfe6ff528f74ac930e4d4c"}
Oct 30 21:00:19.283767 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.280259    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4" event={"ID":"fe91b65f-d75a-4d2a-a526-47ff739cb3ba","Type":"ContainerDied","Data":"4b4f9e9096a73c4927ed7191f476c37ab8dbb1622e0ef710e9dfa4169536d8f2"}
Oct 30 21:00:19.283767 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.280345    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"
Oct 30 21:00:19.283767 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.283245    5967 scope.go:122] "RemoveContainer" containerID="4dab0cd7d5106acf64c7868069cd1dcc27289055badfe6ff528f74ac930e4d4c"
Oct 30 21:00:19.283992 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:19.283886    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv_odh-ai-gateway-infra(5429c2c9-9638-4bd7-8f24-3fd47bf26819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819"
Oct 30 21:00:19.289961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.289165    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j" event={"ID":"9b2cadff-e237-4e97-af50-839f84aa4dee","Type":"ContainerStarted","Data":"4407cdfeca1c6ceb0c3eaae049cb42e82ebcc4af0d2f93c53497c9a1674de158"}
Oct 30 21:00:19.289961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.289215    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j" event={"ID":"9b2cadff-e237-4e97-af50-839f84aa4dee","Type":"ContainerStarted","Data":"54c8a0f0bf581e08e2c9c36072039b97f55c7760f81c6b9a39c66df4b5b15a87"}
Oct 30 21:00:19.293675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.293648    5967 scope.go:122] "RemoveContainer" containerID="11302bd84699b955f4ff5f5c143b27fbd884f741d5f5ad203f8fe958ee20fe4a"
Oct 30 21:00:19.301778 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.300629    5967 generic.go:356] "Generic (PLEG): container finished" podID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerID="ad3b32aa29773095ff1c0becc21f0a5ed487a6eb6f0b58042166c0958b2febdb" exitCode=60
Oct 30 21:00:19.301778 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.300672    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerDied","Data":"ad3b32aa29773095ff1c0becc21f0a5ed487a6eb6f0b58042166c0958b2febdb"}
Oct 30 21:00:19.306541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.306525    5967 scope.go:122] "RemoveContainer" containerID="ad3b32aa29773095ff1c0becc21f0a5ed487a6eb6f0b58042166c0958b2febdb"
Oct 30 21:00:19.306875 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:19.306853    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-9gxtk_odh-ai-gateway-infra(ac0f4093-51ec-4e34-b3c8-9d0b4e316b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27"
Oct 30 21:00:19.322541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.322494    5967 scope.go:122] "RemoveContainer" containerID="8564d90891067412c5bf52fe1e5b6ded636b00be60131ab2ef585f74c200c7c0"
Oct 30 21:00:19.338775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.338717    5967 scope.go:122] "RemoveContainer" containerID="91c8faac1f9acd95bc2fb7502d6c447a4321ed7293e9027aca3ed4457e03c4b2"
Oct 30 21:00:19.355358 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.355262    5967 scope.go:122] "RemoveContainer" containerID="cb69bb88c5c9f551e315fbcb7f2b402bf0e8b0a5d24552a8b6e3bc72e38ef855"
Oct 30 21:00:19.457536 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.457512    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:19.471659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.471631    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:19.632670 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.632644    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk\" (UniqueName: \"kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk\") pod \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") "
Oct 30 21:00:19.632859 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.632786    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca\") pod \"631243a2-00e9-44e7-acb1-28f36973d765\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") "
Oct 30 21:00:19.632936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.632867    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca\") pod \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\" (UID: \"7bc4a21c-3815-40ad-9220-36d162c5f7f1\") "
Oct 30 21:00:19.632936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.632908    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs\" (UniqueName: \"kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs\") pod \"631243a2-00e9-44e7-acb1-28f36973d765\" (UID: \"631243a2-00e9-44e7-acb1-28f36973d765\") "
Oct 30 21:00:19.633397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.633367    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca" pod "631243a2-00e9-44e7-acb1-28f36973d765" (UID: "631243a2-00e9-44e7-acb1-28f36973d765"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.633541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.633449    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca" pod "7bc4a21c-3815-40ad-9220-36d162c5f7f1" (UID: "7bc4a21c-3815-40ad-9220-36d162c5f7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.636181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.636021    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk" pod "7bc4a21c-3815-40ad-9220-36d162c5f7f1" (UID: "7bc4a21c-3815-40ad-9220-36d162c5f7f1"). InnerVolumeSpecName "kube-api-access-7krxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.636181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.636108    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs" pod "631243a2-00e9-44e7-acb1-28f36973d765" (UID: "631243a2-00e9-44e7-acb1-28f36973d765"). InnerVolumeSpecName "kube-api-access-v9bcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.714369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.714349    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"]
Oct 30 21:00:19.733952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.733927    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krxk\" (UniqueName: \"kubernetes.io/projected/7bc4a21c-3815-40ad-9220-36d162c5f7f1-kube-api-access-7krxk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.733952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.733948    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/631243a2-00e9-44e7-acb1-28f36973d765-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.734141 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.733960    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc4a21c-3815-40ad-9220-36d162c5f7f1-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.734141 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.733968    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bcs\" (UniqueName: \"kubernetes.io/projected/631243a2-00e9-44e7-acb1-28f36973d765-kube-api-access-v9bcs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.829686 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.829656    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"]
Oct 30 21:00:19.942905 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:19.942874    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"]
Oct 30 21:00:20.041814 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.040579    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j" podStartSLOduration=3.040561244 podStartE2EDuration="3.040561244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:20.03897907 +0000 UTC m=+2400.639781024" watchObservedRunningTime="2026-10-30 21:00:20.040561244 +0000 UTC m=+2400.641363204"
Oct 30 21:00:20.150632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.150600    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"]
Oct 30 21:00:20.153714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.153598    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-cd2d96-29889900-cxpjw"]
Oct 30 21:00:20.315959 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.315906    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"]
Oct 30 21:00:20.316555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.316533    5967 generic.go:356] "Generic (PLEG): container finished" podID="9b2cadff-e237-4e97-af50-839f84aa4dee" containerID="4407cdfeca1c6ceb0c3eaae049cb42e82ebcc4af0d2f93c53497c9a1674de158" exitCode=0
Oct 30 21:00:20.316653 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.316610    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j" event={"ID":"9b2cadff-e237-4e97-af50-839f84aa4dee","Type":"ContainerDied","Data":"4407cdfeca1c6ceb0c3eaae049cb42e82ebcc4af0d2f93c53497c9a1674de158"}
Oct 30 21:00:20.319774 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.319732    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-65948c-29889900-v8gp4"]
Oct 30 21:00:20.323147 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.323123    5967 generic.go:356] "Generic (PLEG): container finished" podID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerID="2fe10b681d374e532a5072d64feda88442828a22bc5098e3c55f2d794ec25b9d" exitCode=60
Oct 30 21:00:20.323280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.323186    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerDied","Data":"2fe10b681d374e532a5072d64feda88442828a22bc5098e3c55f2d794ec25b9d"}
Oct 30 21:00:20.323280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.323238    5967 scope.go:122] "RemoveContainer" containerID="a7878ee1e6b204c4555b63b85039924b19f4858a7f0e1fd5358990e371392bc3"
Oct 30 21:00:20.323631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.323611    5967 scope.go:122] "RemoveContainer" containerID="2fe10b681d374e532a5072d64feda88442828a22bc5098e3c55f2d794ec25b9d"
Oct 30 21:00:20.326006 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:20.325606    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94_odh-ai-gateway-infra(05a5e5ce-51e3-40a3-bebe-ed036d68704e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e"
Oct 30 21:00:20.327605 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.326795    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"
Oct 30 21:00:20.327605 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.326807    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds" event={"ID":"631243a2-00e9-44e7-acb1-28f36973d765","Type":"ContainerDied","Data":"b43700a91313772f4e2915ed052ae919a8612e4de1d0583e0f1461ac0acb0873"}
Oct 30 21:00:20.381943 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.381915    5967 scope.go:122] "RemoveContainer" containerID="c0e27fc713d904a0ff68bd992b9f1ba92333ce6f429ade6cbb9ee7927e608e66"
Oct 30 21:00:20.397960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.382996    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f" event={"ID":"7bc4a21c-3815-40ad-9220-36d162c5f7f1","Type":"ContainerDied","Data":"f8e6716aeb8b7db8427c9fecd384cc1b8fd288e508aa40b966b541bd124636f5"}
Oct 30 21:00:20.397960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.383120    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"
Oct 30 21:00:20.397960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.393912    5967 generic.go:356] "Generic (PLEG): container finished" podID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerID="d5c31b3074dee2faa7f27da30b343fc7342b89e185a5f2e336f03b865b788aa3" exitCode=60
Oct 30 21:00:20.397960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.393957    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerDied","Data":"d5c31b3074dee2faa7f27da30b343fc7342b89e185a5f2e336f03b865b788aa3"}
Oct 30 21:00:20.397960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.394378    5967 scope.go:122] "RemoveContainer" containerID="d5c31b3074dee2faa7f27da30b343fc7342b89e185a5f2e336f03b865b788aa3"
Oct 30 21:00:20.397960 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:20.394700    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk_odh-ai-gateway-infra(cc6a0f13-7f82-4dca-a2be-f53eb5617cad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad"
Oct 30 21:00:20.424149 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.423660    5967 scope.go:122] "RemoveContainer" containerID="bf239fed917ec216a8ebcbb5f8737541420f077b75ebad3c969b94324351b9bf"
Oct 30 21:00:20.437123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.437100    5967 scope.go:122] "RemoveContainer" containerID="3872f3e3c3912165c82abb6686559138934e8556b1cd5d757bc6b0eb80dceaa8"
Oct 30 21:00:20.548496 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.547148    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:20.586122 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.586100    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca\") pod \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") "
Oct 30 21:00:20.586361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.586347    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6\" (UniqueName: \"kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6\") pod \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\" (UID: \"5429c2c9-9638-4bd7-8f24-3fd47bf26819\") "
Oct 30 21:00:20.587405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.587185    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca" pod "5429c2c9-9638-4bd7-8f24-3fd47bf26819" (UID: "5429c2c9-9638-4bd7-8f24-3fd47bf26819"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.589037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.589011    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6" pod "5429c2c9-9638-4bd7-8f24-3fd47bf26819" (UID: "5429c2c9-9638-4bd7-8f24-3fd47bf26819"). InnerVolumeSpecName "kube-api-access-bxfm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:20.590053 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.590030    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:20.622150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.622129    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:20.687342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.687319    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr\" (UniqueName: \"kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr\") pod \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") "
Oct 30 21:00:20.687447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.687365    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca\") pod \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") "
Oct 30 21:00:20.687525 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.687509    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca\") pod \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\" (UID: \"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27\") "
Oct 30 21:00:20.687624 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.687608    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm\" (UniqueName: \"kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm\") pod \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\" (UID: \"2c34c9e1-2c16-4e5e-b020-94a13d9fd761\") "
Oct 30 21:00:20.687847 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.687822    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca" pod "2c34c9e1-2c16-4e5e-b020-94a13d9fd761" (UID: "2c34c9e1-2c16-4e5e-b020-94a13d9fd761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.687847 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.687844    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca" pod "ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" (UID: "ac0f4093-51ec-4e34-b3c8-9d0b4e316b27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.688113 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.688093    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.688113 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.688112    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfm6\" (UniqueName: \"kubernetes.io/projected/5429c2c9-9638-4bd7-8f24-3fd47bf26819-kube-api-access-bxfm6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.688451 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.688124    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5429c2c9-9638-4bd7-8f24-3fd47bf26819-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.688451 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.688137    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.689607 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.689591    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr" pod "ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" (UID: "ac0f4093-51ec-4e34-b3c8-9d0b4e316b27"). InnerVolumeSpecName "kube-api-access-26hsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:20.689868 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.689846    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm" pod "2c34c9e1-2c16-4e5e-b020-94a13d9fd761" (UID: "2c34c9e1-2c16-4e5e-b020-94a13d9fd761"). InnerVolumeSpecName "kube-api-access-trwvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:20.768531 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.768507    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"]
Oct 30 21:00:20.789256 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.789236    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hsr\" (UniqueName: \"kubernetes.io/projected/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27-kube-api-access-26hsr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.789256 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.789258    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trwvm\" (UniqueName: \"kubernetes.io/projected/2c34c9e1-2c16-4e5e-b020-94a13d9fd761-kube-api-access-trwvm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.932500 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:20.932427    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"]
Oct 30 21:00:21.360084 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.360055    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"]
Oct 30 21:00:21.367322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.367296    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bcef0e-29889900-cpdds"]
Oct 30 21:00:21.410383 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.410343    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv" event={"ID":"5429c2c9-9638-4bd7-8f24-3fd47bf26819","Type":"ContainerDied","Data":"ba1b4adc304d3d93af0133a9f3e0d0fb11b4b982902e4e20381dddab90f1e0fa"}
Oct 30 21:00:21.410522 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.410417    5967 scope.go:122] "RemoveContainer" containerID="4dab0cd7d5106acf64c7868069cd1dcc27289055badfe6ff528f74ac930e4d4c"
Oct 30 21:00:21.410522 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.410358    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"
Oct 30 21:00:21.412247 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.412232    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"
Oct 30 21:00:21.412488 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.412264    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk" event={"ID":"ac0f4093-51ec-4e34-b3c8-9d0b4e316b27","Type":"ContainerDied","Data":"4de442c15d6a5a1ebe52a330ed24570db0927bf821450804bfda2d5e648735c3"}
Oct 30 21:00:21.414466 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.414448    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"
Oct 30 21:00:21.414576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.414458    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m" event={"ID":"2c34c9e1-2c16-4e5e-b020-94a13d9fd761","Type":"ContainerDied","Data":"a9b80ac77aa29ee7412507b5321a4afecd2cf1d61e6a120af6bc145fb2bb55c7"}
Oct 30 21:00:21.432720 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.432611    5967 scope.go:122] "RemoveContainer" containerID="ad3b32aa29773095ff1c0becc21f0a5ed487a6eb6f0b58042166c0958b2febdb"
Oct 30 21:00:21.460802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.460777    5967 scope.go:122] "RemoveContainer" containerID="2f3e547384ef0956f91fb7efd8faade4f613130d1495408a12b2aa7c3e079c3b"
Oct 30 21:00:21.479163 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:21.478626    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429c2c9_9638_4bd7_8f24_3fd47bf26819.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:00:21.486758 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.486713    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"]
Oct 30 21:00:21.498293 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.495734    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5dc13e-29889900-5x22f"]
Oct 30 21:00:21.548512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.548472    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" path="/var/lib/kubelet/pods/45dddb1b-ba6c-4d29-97a7-3c673bea3dc2/volumes"
Oct 30 21:00:21.549492 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.549470    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631243a2-00e9-44e7-acb1-28f36973d765" path="/var/lib/kubelet/pods/631243a2-00e9-44e7-acb1-28f36973d765/volumes"
Oct 30 21:00:21.551194 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.551175    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" path="/var/lib/kubelet/pods/7bc4a21c-3815-40ad-9220-36d162c5f7f1/volumes"
Oct 30 21:00:21.553049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.553030    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" path="/var/lib/kubelet/pods/fe91b65f-d75a-4d2a-a526-47ff739cb3ba/volumes"
Oct 30 21:00:21.625513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.625443    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:21.688460 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.688435    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:21.691753 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.691720    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:21.700159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.700139    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca\") pod \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") "
Oct 30 21:00:21.700272 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.700190    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk\" (UniqueName: \"kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk\") pod \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\" (UID: \"cc6a0f13-7f82-4dca-a2be-f53eb5617cad\") "
Oct 30 21:00:21.700444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.700424    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca" pod "cc6a0f13-7f82-4dca-a2be-f53eb5617cad" (UID: "cc6a0f13-7f82-4dca-a2be-f53eb5617cad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.700649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.700637    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.702384 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.702360    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk" pod "cc6a0f13-7f82-4dca-a2be-f53eb5617cad" (UID: "cc6a0f13-7f82-4dca-a2be-f53eb5617cad"). InnerVolumeSpecName "kube-api-access-fndsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.801575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.801548    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr\" (UniqueName: \"kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr\") pod \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") "
Oct 30 21:00:21.801725 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.801597    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca\") pod \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\" (UID: \"05a5e5ce-51e3-40a3-bebe-ed036d68704e\") "
Oct 30 21:00:21.801725 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.801617    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca\") pod \"9b2cadff-e237-4e97-af50-839f84aa4dee\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") "
Oct 30 21:00:21.801725 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.801639    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8\" (UniqueName: \"kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8\") pod \"9b2cadff-e237-4e97-af50-839f84aa4dee\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") "
Oct 30 21:00:21.801918 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.801850    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca\") pod \"9b2cadff-e237-4e97-af50-839f84aa4dee\" (UID: \"9b2cadff-e237-4e97-af50-839f84aa4dee\") "
Oct 30 21:00:21.802066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.802036    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca" pod "05a5e5ce-51e3-40a3-bebe-ed036d68704e" (UID: "05a5e5ce-51e3-40a3-bebe-ed036d68704e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.802220 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.802195    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca" pod "9b2cadff-e237-4e97-af50-839f84aa4dee" (UID: "9b2cadff-e237-4e97-af50-839f84aa4dee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.802418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.802404    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fndsk\" (UniqueName: \"kubernetes.io/projected/cc6a0f13-7f82-4dca-a2be-f53eb5617cad-kube-api-access-fndsk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.802464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.802421    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.802464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.802431    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a5e5ce-51e3-40a3-bebe-ed036d68704e-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.804018 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.803995    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8" pod "9b2cadff-e237-4e97-af50-839f84aa4dee" (UID: "9b2cadff-e237-4e97-af50-839f84aa4dee"). InnerVolumeSpecName "kube-api-access-gdcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.804319 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.804299    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr" pod "05a5e5ce-51e3-40a3-bebe-ed036d68704e" (UID: "05a5e5ce-51e3-40a3-bebe-ed036d68704e"). InnerVolumeSpecName "kube-api-access-f9ppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.804319 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.804306    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca" pod "9b2cadff-e237-4e97-af50-839f84aa4dee" (UID: "9b2cadff-e237-4e97-af50-839f84aa4dee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:21.834668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.834643    5967 scope.go:122] "RemoveContainer" containerID="be13c20cfefc05e897347710c238557354c5b1dcbcfa96ee11148d9300590876"
Oct 30 21:00:21.847554 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.847520    5967 scope.go:122] "RemoveContainer" containerID="25194597f34dc60fa112c16cb6a92c382c68b7cef874e9a7d97aef3cabc60c8a"
Oct 30 21:00:21.857359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.857341    5967 scope.go:122] "RemoveContainer" containerID="264e69ca5ac4a0ecf20cf814dec8c1da2765b83695b1b2e430432a384d72e920"
Oct 30 21:00:21.869557 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.869522    5967 scope.go:122] "RemoveContainer" containerID="b0a523d1030f4e4a33a012d738d19791ecdb90e4af06175fcd14c351dd64d1d7"
Oct 30 21:00:21.903621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.903594    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ppr\" (UniqueName: \"kubernetes.io/projected/05a5e5ce-51e3-40a3-bebe-ed036d68704e-kube-api-access-f9ppr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.903733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.903628    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2cadff-e237-4e97-af50-839f84aa4dee-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.903733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.903645    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdcv8\" (UniqueName: \"kubernetes.io/projected/9b2cadff-e237-4e97-af50-839f84aa4dee-kube-api-access-gdcv8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.950827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.950799    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"]
Oct 30 21:00:21.957627 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:21.957440    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-9gxtk"]
Oct 30 21:00:22.068486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.068449    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"]
Oct 30 21:00:22.072009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.071985    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-11a0b0-29889900-h9jmv"]
Oct 30 21:00:22.138780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.138699    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"]
Oct 30 21:00:22.143601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.143576    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"]
Oct 30 21:00:22.153610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.153587    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"]
Oct 30 21:00:22.180186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.180162    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"]
Oct 30 21:00:22.185425 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.185402    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ae2d38-29889900-jjk8m"]
Oct 30 21:00:22.441310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.441204    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"
Oct 30 21:00:22.441310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.441228    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j" event={"ID":"9b2cadff-e237-4e97-af50-839f84aa4dee","Type":"ContainerDied","Data":"54c8a0f0bf581e08e2c9c36072039b97f55c7760f81c6b9a39c66df4b5b15a87"}
Oct 30 21:00:22.441310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.441284    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54c8a0f0bf581e08e2c9c36072039b97f55c7760f81c6b9a39c66df4b5b15a87"
Oct 30 21:00:22.445940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.445916    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"
Oct 30 21:00:22.446097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.445939    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94" event={"ID":"05a5e5ce-51e3-40a3-bebe-ed036d68704e","Type":"ContainerDied","Data":"8154cc04110853d11de7aae759620d50b9d0ce7b0b348b7f8aa10ed8a9fa9615"}
Oct 30 21:00:22.446097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.446021    5967 scope.go:122] "RemoveContainer" containerID="2fe10b681d374e532a5072d64feda88442828a22bc5098e3c55f2d794ec25b9d"
Oct 30 21:00:22.448156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.448128    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk" event={"ID":"cc6a0f13-7f82-4dca-a2be-f53eb5617cad","Type":"ContainerDied","Data":"c50b04588802d99edad8d1e03555acd9986abb9869d50d945dc275883e325859"}
Oct 30 21:00:22.448268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.448213    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"
Oct 30 21:00:22.459364 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.459331    5967 scope.go:122] "RemoveContainer" containerID="d5c31b3074dee2faa7f27da30b343fc7342b89e185a5f2e336f03b865b788aa3"
Oct 30 21:00:22.796194 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.796061    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"]
Oct 30 21:00:22.798811 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.798787    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-23491e-29889900-cgdnk"]
Oct 30 21:00:22.914776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.911390    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"]
Oct 30 21:00:22.934775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:22.932077    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-92f229-29889900-mdm94"]
Oct 30 21:00:23.542250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:23.542217    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" path="/var/lib/kubelet/pods/05a5e5ce-51e3-40a3-bebe-ed036d68704e/volumes"
Oct 30 21:00:23.542670 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:23.542656    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" path="/var/lib/kubelet/pods/2c34c9e1-2c16-4e5e-b020-94a13d9fd761/volumes"
Oct 30 21:00:23.543108 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:23.543090    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" path="/var/lib/kubelet/pods/5429c2c9-9638-4bd7-8f24-3fd47bf26819/volumes"
Oct 30 21:00:23.543768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:23.543733    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" path="/var/lib/kubelet/pods/ac0f4093-51ec-4e34-b3c8-9d0b4e316b27/volumes"
Oct 30 21:00:23.544123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:23.544111    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" path="/var/lib/kubelet/pods/cc6a0f13-7f82-4dca-a2be-f53eb5617cad/volumes"
Oct 30 21:00:28.679301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:28.679260    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"]
Oct 30 21:00:30.729423 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.729387    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"]
Oct 30 21:00:30.730035 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730013    5967 cpu_manager.go:414] "containerMap: removing container" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.730035 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730030    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730049    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730054    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730060    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730066    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730073    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730077    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730084    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730089    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730098    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730103    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730111    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730115    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730122    5967 cpu_manager.go:414] "containerMap: removing container" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730126    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730133    5967 cpu_manager.go:414] "containerMap: removing container" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730136    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730142    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9b2cadff-e237-4e97-af50-839f84aa4dee" containerName="revoke-keys"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730147    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2cadff-e237-4e97-af50-839f84aa4dee" containerName="revoke-keys"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730156    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730160    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730165    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730169    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730176    5967 cpu_manager.go:414] "containerMap: removing container" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730180    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730188    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730192    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730198    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.730195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730203    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730210    5967 cpu_manager.go:414] "containerMap: removing container" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730215    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730223    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730227    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730234    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730238    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730247    5967 cpu_manager.go:414] "containerMap: removing container" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730251    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730259    5967 cpu_manager.go:414] "containerMap: removing container" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730263    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730269    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730273    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730281    5967 cpu_manager.go:414] "containerMap: removing container" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730284    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730299    5967 cpu_manager.go:414] "containerMap: removing container" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730303    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730402    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730408    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730415    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730419    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730424    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730431    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730436    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730441    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730447    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730451    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730456    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730462    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730467    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730472    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2cadff-e237-4e97-af50-839f84aa4dee" containerName="revoke-keys"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730478    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c34c9e1-2c16-4e5e-b020-94a13d9fd761" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730483    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730488    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730494    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730499    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730503    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730510    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730515    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0f4093-51ec-4e34-b3c8-9d0b4e316b27" containerName="cleanup"
Oct 30 21:00:30.731054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.730521    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.735780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.735762    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.747922 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.747898    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgql\" (UniqueName: \"kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.748025 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.747950    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.748152 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.748129    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.790622 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.790597    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"]
Oct 30 21:00:30.836066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.836038    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:30.836183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.836114    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:30.836183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.836117    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:00:30.837023 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.836999    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:30.849706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.849678    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.849840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.849826    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgql\" (UniqueName: \"kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.849918 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.849902    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.850418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.850400    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.852355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.852334    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.906794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.906768    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"]
Oct 30 21:00:30.907314 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907302    5967 cpu_manager.go:414] "containerMap: removing container" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.907353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907316    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.907353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907330    5967 cpu_manager.go:414] "containerMap: removing container" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.907353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907335    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.907454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907362    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.907454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907367    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc4a21c-3815-40ad-9220-36d162c5f7f1" containerName="cleanup"
Oct 30 21:00:30.907454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907377    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.907454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907381    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.907454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907395    5967 cpu_manager.go:414] "containerMap: removing container" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.907454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907399    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631243a2-00e9-44e7-acb1-28f36973d765" containerName="cleanup"
Oct 30 21:00:30.907630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907483    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6a0f13-7f82-4dca-a2be-f53eb5617cad" containerName="cleanup"
Oct 30 21:00:30.907630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907492    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dddb1b-ba6c-4d29-97a7-3c673bea3dc2" containerName="cleanup"
Oct 30 21:00:30.907630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907502    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe91b65f-d75a-4d2a-a526-47ff739cb3ba" containerName="cleanup"
Oct 30 21:00:30.907630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907514    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a5e5ce-51e3-40a3-bebe-ed036d68704e" containerName="cleanup"
Oct 30 21:00:30.907630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.907519    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5429c2c9-9638-4bd7-8f24-3fd47bf26819" containerName="cleanup"
Oct 30 21:00:30.908027 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.908010    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgql\" (UniqueName: \"kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql\") pod \"maas-api-revoke-sub-a812710b-d2xgn\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:30.911102 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.911087    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:30.950710 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.950652    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:30.950838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.950719    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:30.950904 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.950885    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ffs\" (UniqueName: \"kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:30.953793 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.953769    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"]
Oct 30 21:00:30.953964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.953945    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" podUID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" containerName="discovery" containerID="cri-o://cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29" gracePeriod=30
Oct 30 21:00:30.971259 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:30.971234    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"]
Oct 30 21:00:31.049830 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.049803    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:31.052055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.052032    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.052170 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.052068    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.052237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.052220    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ffs\" (UniqueName: \"kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.053031 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.053011    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.059160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.059139    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.109349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.109324    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ffs\" (UniqueName: \"kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs\") pod \"maas-api-revoke-sub-b5576b5b-ktjrq\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.232778 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.232670    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:31.255830 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.255807    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 21:00:31.356793 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.356727    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.356959 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.356823    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.357019 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.356976    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.357057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.357026    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.357105 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.357058    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.357141 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.357099    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.357210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.357183    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token\") pod \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\" (UID: \"1f1fa104-3fab-4a1c-aff9-b286df34e1af\") "
Oct 30 21:00:31.357267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.357216    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:31.357782 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.357712    5967 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-ca-configmap\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.359168 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.359143    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:31.359283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.359251    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:31.359600 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.359585    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:31.359704 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.359690    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "kube-api-access-mrvl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:31.359848 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.359826    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:31.359909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.359894    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token" pod "1f1fa104-3fab-4a1c-aff9-b286df34e1af" (UID: "1f1fa104-3fab-4a1c-aff9-b286df34e1af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:31.443389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.443195    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"]
Oct 30 21:00:31.446992 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:31.446613    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9900dc7_a20c_448c_a2ae_f33e1ccd4b0b.slice/crio-bfee890fbfecbdd84e3d0fdffd83c9d4662c8de3e9b5e5aaa3f415820f4f33ee WatchSource:0}: Error finding container bfee890fbfecbdd84e3d0fdffd83c9d4662c8de3e9b5e5aaa3f415820f4f33ee: Status 404 returned error can't find the container with id bfee890fbfecbdd84e3d0fdffd83c9d4662c8de3e9b5e5aaa3f415820f4f33ee
Oct 30 21:00:31.458677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.458655    5967 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-cacerts\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.458772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.458683    5967 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-kubeconfig\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.458772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.458699    5967 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-csr-dns-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.458772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.458711    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrvl6\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-kube-api-access-mrvl6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.458772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.458720    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1fa104-3fab-4a1c-aff9-b286df34e1af-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.458772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.458728    5967 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1fa104-3fab-4a1c-aff9-b286df34e1af-local-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.521697 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.521611    5967 generic.go:356] "Generic (PLEG): container finished" podID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" containerID="cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29" exitCode=0
Oct 30 21:00:31.521697 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.521670    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"
Oct 30 21:00:31.521935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.521705    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" event={"ID":"1f1fa104-3fab-4a1c-aff9-b286df34e1af","Type":"ContainerDied","Data":"cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29"}
Oct 30 21:00:31.521935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.521787    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw" event={"ID":"1f1fa104-3fab-4a1c-aff9-b286df34e1af","Type":"ContainerDied","Data":"7001f19145338bf44cc7e053b91157db2cccc8bcb29b81b7f3d6831eb0f25b16"}
Oct 30 21:00:31.521935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.521828    5967 scope.go:122] "RemoveContainer" containerID="cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29"
Oct 30 21:00:31.533599 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.533562    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn" event={"ID":"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b","Type":"ContainerStarted","Data":"bfee890fbfecbdd84e3d0fdffd83c9d4662c8de3e9b5e5aaa3f415820f4f33ee"}
Oct 30 21:00:31.545046 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.545027    5967 scope.go:122] "RemoveContainer" containerID="cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29"
Oct 30 21:00:31.545378 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:31.545351    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29\": container with ID starting with cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29 not found: ID does not exist" containerID="cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29"
Oct 30 21:00:31.545454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.545384    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29"} err="failed to get container status \"cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29\": rpc error: code = NotFound desc = could not find container \"cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29\": container with ID starting with cefcee3beb7b257e9557b7d53f9a8b9c84d105484dc92abfb93ca54c82f21e29 not found: ID does not exist"
Oct 30 21:00:31.604989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.604961    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"]
Oct 30 21:00:31.606886 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:00:31.606858    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74847b1_cc9b_40d8_a46e_265a0c1350dd.slice/crio-6dfc2f5af33575176ef625a361d7892e1e933cc0e8ed36433d9fa917b369c7ea WatchSource:0}: Error finding container 6dfc2f5af33575176ef625a361d7892e1e933cc0e8ed36433d9fa917b369c7ea: Status 404 returned error can't find the container with id 6dfc2f5af33575176ef625a361d7892e1e933cc0e8ed36433d9fa917b369c7ea
Oct 30 21:00:31.685762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.685720    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"]
Oct 30 21:00:31.690810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:31.690786    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw"]
Oct 30 21:00:31.981245 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:31.981212    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:00:32.543510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:32.543488    5967 generic.go:356] "Generic (PLEG): container finished" podID="e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" containerID="96d87f53e3cc9f27b2534efeff037743c20daa799fc6eaa8f9b64661c28373c0" exitCode=0
Oct 30 21:00:32.543652 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:32.543571    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn" event={"ID":"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b","Type":"ContainerDied","Data":"96d87f53e3cc9f27b2534efeff037743c20daa799fc6eaa8f9b64661c28373c0"}
Oct 30 21:00:32.545049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:32.545028    5967 generic.go:356] "Generic (PLEG): container finished" podID="c74847b1-cc9b-40d8-a46e-265a0c1350dd" containerID="1acbf2c917f9854746a5f498ab56f5a95b9abd4976105a90bede938ae5c518aa" exitCode=0
Oct 30 21:00:32.545155 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:32.545102    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq" event={"ID":"c74847b1-cc9b-40d8-a46e-265a0c1350dd","Type":"ContainerDied","Data":"1acbf2c917f9854746a5f498ab56f5a95b9abd4976105a90bede938ae5c518aa"}
Oct 30 21:00:32.545155 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:32.545146    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq" event={"ID":"c74847b1-cc9b-40d8-a46e-265a0c1350dd","Type":"ContainerStarted","Data":"6dfc2f5af33575176ef625a361d7892e1e933cc0e8ed36433d9fa917b369c7ea"}
Oct 30 21:00:33.543686 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.543655    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" path="/var/lib/kubelet/pods/1f1fa104-3fab-4a1c-aff9-b286df34e1af/volumes"
Oct 30 21:00:33.679853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.679814    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" containerName="payload-processing" containerID="cri-o://a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f" gracePeriod=25
Oct 30 21:00:33.722301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.722280    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:33.724650 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.724631    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:33.785062 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785038    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca\") pod \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") "
Oct 30 21:00:33.785177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785128    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca\") pod \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") "
Oct 30 21:00:33.785177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785170    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca\") pod \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") "
Oct 30 21:00:33.785260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785196    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql\" (UniqueName: \"kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql\") pod \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\" (UID: \"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b\") "
Oct 30 21:00:33.785410 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785388    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs\" (UniqueName: \"kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs\") pod \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") "
Oct 30 21:00:33.785507 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785388    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca" pod "e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" (UID: "e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:33.785507 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785471    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca\") pod \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\" (UID: \"c74847b1-cc9b-40d8-a46e-265a0c1350dd\") "
Oct 30 21:00:33.785630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.785454    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca" pod "c74847b1-cc9b-40d8-a46e-265a0c1350dd" (UID: "c74847b1-cc9b-40d8-a46e-265a0c1350dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:33.786253 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.786229    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:33.786349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.786259    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:33.787405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.787377    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca" pod "e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" (UID: "e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:33.787405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.787394    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql" pod "e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" (UID: "e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b"). InnerVolumeSpecName "kube-api-access-fvgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:33.787542 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.787453    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs" pod "c74847b1-cc9b-40d8-a46e-265a0c1350dd" (UID: "c74847b1-cc9b-40d8-a46e-265a0c1350dd"). InnerVolumeSpecName "kube-api-access-g8ffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:33.787542 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.787498    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca" pod "c74847b1-cc9b-40d8-a46e-265a0c1350dd" (UID: "c74847b1-cc9b-40d8-a46e-265a0c1350dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:33.887345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.887315    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:33.887345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.887342    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgql\" (UniqueName: \"kubernetes.io/projected/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b-kube-api-access-fvgql\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:33.887345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.887353    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8ffs\" (UniqueName: \"kubernetes.io/projected/c74847b1-cc9b-40d8-a46e-265a0c1350dd-kube-api-access-g8ffs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:33.887563 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:33.887362    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c74847b1-cc9b-40d8-a46e-265a0c1350dd-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:34.559615 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:34.559582    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"
Oct 30 21:00:34.560106 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:34.559630    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn" event={"ID":"e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b","Type":"ContainerDied","Data":"bfee890fbfecbdd84e3d0fdffd83c9d4662c8de3e9b5e5aaa3f415820f4f33ee"}
Oct 30 21:00:34.560106 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:34.559674    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfee890fbfecbdd84e3d0fdffd83c9d4662c8de3e9b5e5aaa3f415820f4f33ee"
Oct 30 21:00:34.561189 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:34.561163    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"
Oct 30 21:00:34.561290 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:34.561196    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq" event={"ID":"c74847b1-cc9b-40d8-a46e-265a0c1350dd","Type":"ContainerDied","Data":"6dfc2f5af33575176ef625a361d7892e1e933cc0e8ed36433d9fa917b369c7ea"}
Oct 30 21:00:34.561290 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:34.561233    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dfc2f5af33575176ef625a361d7892e1e933cc0e8ed36433d9fa917b369c7ea"
Oct 30 21:00:37.165926 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.165879    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main" containerID="cri-o://06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509" gracePeriod=45
Oct 30 21:00:37.177161 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.177132    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" containerID="cri-o://1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a" gracePeriod=45
Oct 30 21:00:37.244558 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.244525    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 21:00:37.439437 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.439411    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 21:00:37.460885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.460865    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 21:00:37.477513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.477491    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main" containerID="cri-o://92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1" gracePeriod=45
Oct 30 21:00:37.534423 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534398    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir\") pod \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") "
Oct 30 21:00:37.534559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534437    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs\") pod \"3b9a740f-ff20-45ba-a128-24f890436aa4\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") "
Oct 30 21:00:37.534559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534468    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv\" (UniqueName: \"kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv\") pod \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") "
Oct 30 21:00:37.534559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534488    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm\") pod \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") "
Oct 30 21:00:37.534559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534522    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home\") pod \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") "
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534584    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache\") pod \"3b9a740f-ff20-45ba-a128-24f890436aa4\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") "
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534626    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm\" (UniqueName: \"kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm\") pod \"3b9a740f-ff20-45ba-a128-24f890436aa4\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") "
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534661    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs\") pod \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") "
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534682    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir" pod "4c5b9238-e7d3-4fc3-b731-aed5b4df138d" (UID: "4c5b9238-e7d3-4fc3-b731-aed5b4df138d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534695    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir\") pod \"3b9a740f-ff20-45ba-a128-24f890436aa4\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") "
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534720    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home\") pod \"3b9a740f-ff20-45ba-a128-24f890436aa4\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") "
Oct 30 21:00:37.534819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534781    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache\") pod \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\" (UID: \"4c5b9238-e7d3-4fc3-b731-aed5b4df138d\") "
Oct 30 21:00:37.535403 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534880    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm\") pod \"3b9a740f-ff20-45ba-a128-24f890436aa4\" (UID: \"3b9a740f-ff20-45ba-a128-24f890436aa4\") "
Oct 30 21:00:37.535540 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.535519    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.535601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.534878    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home" pod "4c5b9238-e7d3-4fc3-b731-aed5b4df138d" (UID: "4c5b9238-e7d3-4fc3-b731-aed5b4df138d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.535659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.535595    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache" pod "3b9a740f-ff20-45ba-a128-24f890436aa4" (UID: "3b9a740f-ff20-45ba-a128-24f890436aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.535659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.535605    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache" pod "4c5b9238-e7d3-4fc3-b731-aed5b4df138d" (UID: "4c5b9238-e7d3-4fc3-b731-aed5b4df138d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.535808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.535663    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir" pod "3b9a740f-ff20-45ba-a128-24f890436aa4" (UID: "3b9a740f-ff20-45ba-a128-24f890436aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.535869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.535818    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home" pod "3b9a740f-ff20-45ba-a128-24f890436aa4" (UID: "3b9a740f-ff20-45ba-a128-24f890436aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.537147 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.537125    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm" pod "4c5b9238-e7d3-4fc3-b731-aed5b4df138d" (UID: "4c5b9238-e7d3-4fc3-b731-aed5b4df138d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.537289 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.537267    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs" pod "3b9a740f-ff20-45ba-a128-24f890436aa4" (UID: "3b9a740f-ff20-45ba-a128-24f890436aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:37.537345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.537289    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv" pod "4c5b9238-e7d3-4fc3-b731-aed5b4df138d" (UID: "4c5b9238-e7d3-4fc3-b731-aed5b4df138d"). InnerVolumeSpecName "kube-api-access-4kjrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:37.537799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.537784    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs" pod "4c5b9238-e7d3-4fc3-b731-aed5b4df138d" (UID: "4c5b9238-e7d3-4fc3-b731-aed5b4df138d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:37.538083 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.538063    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm" pod "3b9a740f-ff20-45ba-a128-24f890436aa4" (UID: "3b9a740f-ff20-45ba-a128-24f890436aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.538131 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.538110    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm" pod "3b9a740f-ff20-45ba-a128-24f890436aa4" (UID: "3b9a740f-ff20-45ba-a128-24f890436aa4"). InnerVolumeSpecName "kube-api-access-v9zpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:37.586011 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.585977    5967 generic.go:356] "Generic (PLEG): container finished" podID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerID="1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a" exitCode=0
Oct 30 21:00:37.586177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.586034    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"
Oct 30 21:00:37.586177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.586034    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" event={"ID":"4c5b9238-e7d3-4fc3-b731-aed5b4df138d","Type":"ContainerDied","Data":"1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a"}
Oct 30 21:00:37.586177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.586095    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7" event={"ID":"4c5b9238-e7d3-4fc3-b731-aed5b4df138d","Type":"ContainerDied","Data":"fb3b1744903ea9d570dad76cb56a3e74a6e8a9a7b2e641027d32765e2cdc22e6"}
Oct 30 21:00:37.586177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.586121    5967 scope.go:122] "RemoveContainer" containerID="1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a"
Oct 30 21:00:37.589042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.589014    5967 generic.go:356] "Generic (PLEG): container finished" podID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerID="06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509" exitCode=0
Oct 30 21:00:37.589151 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.589066    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"
Oct 30 21:00:37.589151 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.589072    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" event={"ID":"3b9a740f-ff20-45ba-a128-24f890436aa4","Type":"ContainerDied","Data":"06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509"}
Oct 30 21:00:37.589151 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.589104    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2" event={"ID":"3b9a740f-ff20-45ba-a128-24f890436aa4","Type":"ContainerDied","Data":"0b7bcadeba41951758a5d5c04ed6ff895a35abeeb913b38ba3e5745e6acfb73a"}
Oct 30 21:00:37.599063 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.598997    5967 scope.go:122] "RemoveContainer" containerID="1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a"
Oct 30 21:00:37.599408 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:37.599271    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a\": container with ID starting with 1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a not found: ID does not exist" containerID="1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a"
Oct 30 21:00:37.599408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.599302    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a"} err="failed to get container status \"1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a\": rpc error: code = NotFound desc = could not find container \"1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a\": container with ID starting with 1dec1b4c9164dc303ad1bf20045877365690d0cf74c6cfaa257af0f42c1dc80a not found: ID does not exist"
Oct 30 21:00:37.599408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.599324    5967 scope.go:122] "RemoveContainer" containerID="06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509"
Oct 30 21:00:37.609788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.609767    5967 scope.go:122] "RemoveContainer" containerID="06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509"
Oct 30 21:00:37.610063 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:37.610045    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509\": container with ID starting with 06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509 not found: ID does not exist" containerID="06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509"
Oct 30 21:00:37.610149 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.610070    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509"} err="failed to get container status \"06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509\": rpc error: code = NotFound desc = could not find container \"06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509\": container with ID starting with 06c9649aaa478e0d7d553e70b056a5559861c30d6f1e5e9f72cf22a9f3054509 not found: ID does not exist"
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636916    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9a740f-ff20-45ba-a128-24f890436aa4-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636947    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjrv\" (UniqueName: \"kubernetes.io/projected/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-kube-api-access-4kjrv\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636960    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636967    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636976    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636985    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zpm\" (UniqueName: \"kubernetes.io/projected/3b9a740f-ff20-45ba-a128-24f890436aa4-kube-api-access-v9zpm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.636995    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.637007    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.637014    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.637021    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5b9238-e7d3-4fc3-b731-aed5b4df138d-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.637049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.637030    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9a740f-ff20-45ba-a128-24f890436aa4-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.666978 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.666904    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:37.732657 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.732635    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 21:00:37.786206 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.786174    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"]
Oct 30 21:00:37.788728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.788701    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/premium-w4-8e2fc1-kserve-847dcb587-c5sl7"]
Oct 30 21:00:37.839510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839483    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir\") pod \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") "
Oct 30 21:00:37.839645 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839537    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm\") pod \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") "
Oct 30 21:00:37.839645 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839601    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4\" (UniqueName: \"kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4\") pod \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") "
Oct 30 21:00:37.839777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839662    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache\") pod \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") "
Oct 30 21:00:37.839777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839694    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs\") pod \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") "
Oct 30 21:00:37.839877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839781    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir" pod "73da3e8a-bc34-4aa6-91e2-61f34e6271dc" (UID: "73da3e8a-bc34-4aa6-91e2-61f34e6271dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.839877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839842    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home\") pod \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\" (UID: \"73da3e8a-bc34-4aa6-91e2-61f34e6271dc\") "
Oct 30 21:00:37.840033 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.839942    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache" pod "73da3e8a-bc34-4aa6-91e2-61f34e6271dc" (UID: "73da3e8a-bc34-4aa6-91e2-61f34e6271dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.840033 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.840007    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home" pod "73da3e8a-bc34-4aa6-91e2-61f34e6271dc" (UID: "73da3e8a-bc34-4aa6-91e2-61f34e6271dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.840409 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.840390    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.840500 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.840414    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.840500 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.840427    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.841868 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.841845    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm" pod "73da3e8a-bc34-4aa6-91e2-61f34e6271dc" (UID: "73da3e8a-bc34-4aa6-91e2-61f34e6271dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:37.841962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.841866    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs" pod "73da3e8a-bc34-4aa6-91e2-61f34e6271dc" (UID: "73da3e8a-bc34-4aa6-91e2-61f34e6271dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:37.841962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.841894    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4" pod "73da3e8a-bc34-4aa6-91e2-61f34e6271dc" (UID: "73da3e8a-bc34-4aa6-91e2-61f34e6271dc"). InnerVolumeSpecName "kube-api-access-2nfj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:37.892852 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.892825    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"]
Oct 30 21:00:37.897134 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.897113    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-w4-8e2fc1-kserve-886c4b4db-psnh2"]
Oct 30 21:00:37.941428 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.941380    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.941428 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.941400    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfj4\" (UniqueName: \"kubernetes.io/projected/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-kube-api-access-2nfj4\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.941428 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:37.941412    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73da3e8a-bc34-4aa6-91e2-61f34e6271dc-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:38.596920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.596884    5967 generic.go:356] "Generic (PLEG): container finished" podID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerID="92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1" exitCode=0
Oct 30 21:00:38.597355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.596926    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"
Oct 30 21:00:38.597355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.596965    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" event={"ID":"73da3e8a-bc34-4aa6-91e2-61f34e6271dc","Type":"ContainerDied","Data":"92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1"}
Oct 30 21:00:38.597355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.597005    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd" event={"ID":"73da3e8a-bc34-4aa6-91e2-61f34e6271dc","Type":"ContainerDied","Data":"d73ac614a13f448828b302b300eaf8ccaa96039bd8c3370152dddab251c5a071"}
Oct 30 21:00:38.597355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.597024    5967 scope.go:122] "RemoveContainer" containerID="92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1"
Oct 30 21:00:38.609815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.609666    5967 scope.go:122] "RemoveContainer" containerID="92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1"
Oct 30 21:00:38.610311 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:38.610267    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1\": container with ID starting with 92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1 not found: ID does not exist" containerID="92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1"
Oct 30 21:00:38.610430 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.610320    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1"} err="failed to get container status \"92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1\": rpc error: code = NotFound desc = could not find container \"92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1\": container with ID starting with 92faca3c411d49145df44ed13f2a2cccd054b6d43beec22fb8af9105014af9a1 not found: ID does not exist"
Oct 30 21:00:38.761418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.761384    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"]
Oct 30 21:00:38.810262 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:38.810231    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e2fc1/distinct-2-w4-8e2fc1-kserve-5889db975c-cz5cd"]
Oct 30 21:00:39.544622 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:39.544588    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" path="/var/lib/kubelet/pods/3b9a740f-ff20-45ba-a128-24f890436aa4/volumes"
Oct 30 21:00:39.545113 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:39.545092    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" path="/var/lib/kubelet/pods/4c5b9238-e7d3-4fc3-b731-aed5b4df138d/volumes"
Oct 30 21:00:39.545538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:39.545519    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" path="/var/lib/kubelet/pods/73da3e8a-bc34-4aa6-91e2-61f34e6271dc/volumes"
Oct 30 21:00:42.543849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.543816    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"]
Oct 30 21:00:42.544434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544415    5967 cpu_manager.go:414] "containerMap: removing container" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main"
Oct 30 21:00:42.544499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544438    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main"
Oct 30 21:00:42.544499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544463    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main"
Oct 30 21:00:42.544499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544471    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main"
Oct 30 21:00:42.544499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544490    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main"
Oct 30 21:00:42.544499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544498    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main"
Oct 30 21:00:42.544668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544525    5967 cpu_manager.go:414] "containerMap: removing container" podUID="c74847b1-cc9b-40d8-a46e-265a0c1350dd" containerName="revoke-keys"
Oct 30 21:00:42.544668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544533    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74847b1-cc9b-40d8-a46e-265a0c1350dd" containerName="revoke-keys"
Oct 30 21:00:42.544668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544554    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" containerName="revoke-keys"
Oct 30 21:00:42.544668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544560    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" containerName="revoke-keys"
Oct 30 21:00:42.544668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544590    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" containerName="discovery"
Oct 30 21:00:42.544668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544596    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" containerName="discovery"
Oct 30 21:00:42.544894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544764    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da3e8a-bc34-4aa6-91e2-61f34e6271dc" containerName="main"
Oct 30 21:00:42.544894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544782    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5b9238-e7d3-4fc3-b731-aed5b4df138d" containerName="main"
Oct 30 21:00:42.544894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544792    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9a740f-ff20-45ba-a128-24f890436aa4" containerName="main"
Oct 30 21:00:42.544894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544807    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74847b1-cc9b-40d8-a46e-265a0c1350dd" containerName="revoke-keys"
Oct 30 21:00:42.544894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544817    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" containerName="revoke-keys"
Oct 30 21:00:42.544894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.544830    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1fa104-3fab-4a1c-aff9-b286df34e1af" containerName="discovery"
Oct 30 21:00:42.548582 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.548563    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.652480 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.652454    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:42.652789 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.652729    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:42.652789 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.652777    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:00:42.653006 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.652730    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:42.655772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.655735    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"]
Oct 30 21:00:42.692663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.692636    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.692783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.692727    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.692884 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.692865    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwc5z\" (UniqueName: \"kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.794224 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.794166    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.794329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.794226    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.794329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.794269    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwc5z\" (UniqueName: \"kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.794797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.794774    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.796652 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.796627    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.850922 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.850902    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwc5z\" (UniqueName: \"kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z\") pod \"maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:42.860707 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:42.860688    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:43.360008 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:43.357902    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"]
Oct 30 21:00:43.665136 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:43.665031    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd" event={"ID":"3a696167-db92-4f40-b647-a1480edec193","Type":"ContainerStarted","Data":"f80a7a9175113b890a7ed4152cc7c6ade01a3d42b325d3f730d7cf1509220011"}
Oct 30 21:00:44.707763 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:44.704724    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd" event={"ID":"3a696167-db92-4f40-b647-a1480edec193","Type":"ContainerStarted","Data":"e1c2644b10ea9bd763ba5f44b33fbc4634e70595b3889eaf624f4cb6387d9169"}
Oct 30 21:00:45.712989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:45.712958    5967 generic.go:356] "Generic (PLEG): container finished" podID="3a696167-db92-4f40-b647-a1480edec193" containerID="e1c2644b10ea9bd763ba5f44b33fbc4634e70595b3889eaf624f4cb6387d9169" exitCode=0
Oct 30 21:00:45.713380 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:45.713026    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd" event={"ID":"3a696167-db92-4f40-b647-a1480edec193","Type":"ContainerDied","Data":"e1c2644b10ea9bd763ba5f44b33fbc4634e70595b3889eaf624f4cb6387d9169"}
Oct 30 21:00:46.888104 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.888077    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:46.918651 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.918626    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca\") pod \"3a696167-db92-4f40-b647-a1480edec193\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") "
Oct 30 21:00:46.918829 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.918804    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z\" (UniqueName: \"kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z\") pod \"3a696167-db92-4f40-b647-a1480edec193\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") "
Oct 30 21:00:46.918894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.918871    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca\") pod \"3a696167-db92-4f40-b647-a1480edec193\" (UID: \"3a696167-db92-4f40-b647-a1480edec193\") "
Oct 30 21:00:46.919017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.918995    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca" pod "3a696167-db92-4f40-b647-a1480edec193" (UID: "3a696167-db92-4f40-b647-a1480edec193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:46.919279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.919263    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a696167-db92-4f40-b647-a1480edec193-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:46.920847 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.920824    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z" pod "3a696167-db92-4f40-b647-a1480edec193" (UID: "3a696167-db92-4f40-b647-a1480edec193"). InnerVolumeSpecName "kube-api-access-qwc5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:46.920970 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:46.920868    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca" pod "3a696167-db92-4f40-b647-a1480edec193" (UID: "3a696167-db92-4f40-b647-a1480edec193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:47.020249 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:47.020184    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwc5z\" (UniqueName: \"kubernetes.io/projected/3a696167-db92-4f40-b647-a1480edec193-kube-api-access-qwc5z\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:47.020249 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:47.020212    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a696167-db92-4f40-b647-a1480edec193-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:47.664300 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:47.664266    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:47.727835 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:47.727804    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd" event={"ID":"3a696167-db92-4f40-b647-a1480edec193","Type":"ContainerDied","Data":"f80a7a9175113b890a7ed4152cc7c6ade01a3d42b325d3f730d7cf1509220011"}
Oct 30 21:00:47.727995 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:47.727851    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80a7a9175113b890a7ed4152cc7c6ade01a3d42b325d3f730d7cf1509220011"
Oct 30 21:00:47.727995 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:47.727849    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"
Oct 30 21:00:53.928893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:53.928869    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 21:00:53.991323 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:53.991301    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume\") pod \"ac979096-2a91-4bdb-9e84-324839fcb845\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") "
Oct 30 21:00:53.991465 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:53.991424    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd\" (UniqueName: \"kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd\") pod \"ac979096-2a91-4bdb-9e84-324839fcb845\" (UID: \"ac979096-2a91-4bdb-9e84-324839fcb845\") "
Oct 30 21:00:53.991625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:53.991603    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume" pod "ac979096-2a91-4bdb-9e84-324839fcb845" (UID: "ac979096-2a91-4bdb-9e84-324839fcb845"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:53.991720 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:53.991708    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac979096-2a91-4bdb-9e84-324839fcb845-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:53.993466 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:53.993444    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd" pod "ac979096-2a91-4bdb-9e84-324839fcb845" (UID: "ac979096-2a91-4bdb-9e84-324839fcb845"). InnerVolumeSpecName "kube-api-access-t9gjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:54.093097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.093063    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gjd\" (UniqueName: \"kubernetes.io/projected/ac979096-2a91-4bdb-9e84-324839fcb845-kube-api-access-t9gjd\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.774706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.774672    5967 generic.go:356] "Generic (PLEG): container finished" podID="ac979096-2a91-4bdb-9e84-324839fcb845" containerID="a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f" exitCode=0
Oct 30 21:00:54.774917 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.774732    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"
Oct 30 21:00:54.774917 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.774735    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" event={"ID":"ac979096-2a91-4bdb-9e84-324839fcb845","Type":"ContainerDied","Data":"a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f"}
Oct 30 21:00:54.774917 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.774834    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8" event={"ID":"ac979096-2a91-4bdb-9e84-324839fcb845","Type":"ContainerDied","Data":"72a28aaf7adbebf6abc418e72738ef6787e26cfea13ffa9c489faa53ac324f77"}
Oct 30 21:00:54.774917 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.774862    5967 scope.go:122] "RemoveContainer" containerID="a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f"
Oct 30 21:00:54.788933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.788908    5967 scope.go:122] "RemoveContainer" containerID="a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f"
Oct 30 21:00:54.789230 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:00:54.789196    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f\": container with ID starting with a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f not found: ID does not exist" containerID="a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f"
Oct 30 21:00:54.789344 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.789227    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f"} err="failed to get container status \"a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f\": rpc error: code = NotFound desc = could not find container \"a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f\": container with ID starting with a9a7ed2bfd976f103546717d84e4fc694cd836f7e5fbba2ea85f5c0c8f95804f not found: ID does not exist"
Oct 30 21:00:54.937527 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.937493    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"]
Oct 30 21:00:54.958315 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:54.958290    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8"]
Oct 30 21:00:55.546419 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:00:55.546389    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" path="/var/lib/kubelet/pods/ac979096-2a91-4bdb-9e84-324839fcb845/volumes"
Oct 30 21:01:01.979379 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:01:01.979305    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:01:12.531712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:12.531679    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"]
Oct 30 21:01:12.580466 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:12.580439    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65948c-lgth8"]
Oct 30 21:01:13.543157 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:13.543123    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32d5eb8-87b1-453b-a8ea-ecbbddb99a08" path="/var/lib/kubelet/pods/c32d5eb8-87b1-453b-a8ea-ecbbddb99a08/volumes"
Oct 30 21:01:21.212692 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.212657    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"]
Oct 30 21:01:21.213261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.213245    5967 cpu_manager.go:414] "containerMap: removing container" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" containerName="payload-processing"
Oct 30 21:01:21.213306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.213264    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" containerName="payload-processing"
Oct 30 21:01:21.213306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.213298    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3a696167-db92-4f40-b647-a1480edec193" containerName="revoke-keys"
Oct 30 21:01:21.213306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.213304    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a696167-db92-4f40-b647-a1480edec193" containerName="revoke-keys"
Oct 30 21:01:21.213407 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.213384    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a696167-db92-4f40-b647-a1480edec193" containerName="revoke-keys"
Oct 30 21:01:21.213407 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.213398    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac979096-2a91-4bdb-9e84-324839fcb845" containerName="payload-processing"
Oct 30 21:01:21.216458 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.216445    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.322825 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.322793    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.322868    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.322898    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.322931    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323158 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.323028    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323158 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.323081    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323158 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.323098    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323323 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.323249    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvp7\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323323 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.323312    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.323682 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.323636    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"]
Oct 30 21:01:21.369693 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.369666    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2fa881-openshift-default-dockercfg-mszxc\""
Oct 30 21:01:21.424584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424547    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.424777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424593    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.424777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424640    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvp7\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.424777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424677    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.424777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424726    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424803    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424827    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424863    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.424949    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425248 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.425143    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.425334    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.425403    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.425428    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.425667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.425643    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.427892 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.427862    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.428338 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.428320    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.482790 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.482763    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.483054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.483037    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvp7\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7\") pod \"e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") " pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.529883 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.529863    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:21.963398 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.963372    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"]
Oct 30 21:01:21.964673 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:01:21.964647    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9a8bbd_bb10_464a_a543_6ebc193166a0.slice/crio-3f10c1713c87d291ce66f9f2aea7d71f1f1ab27b9038934ad55c573fce179eee WatchSource:0}: Error finding container 3f10c1713c87d291ce66f9f2aea7d71f1f1ab27b9038934ad55c573fce179eee: Status 404 returned error can't find the container with id 3f10c1713c87d291ce66f9f2aea7d71f1f1ab27b9038934ad55c573fce179eee
Oct 30 21:01:21.967057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.967020    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:01:21.967166 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.967088    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:01:21.967166 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.967117    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:01:21.977271 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.977248    5967 scope.go:122] "RemoveContainer" containerID="977cc3c59979b9d8821931dca5f8a5ffe43ad94d8e0d39e79c18b7dfd820b269"
Oct 30 21:01:21.986577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:21.986559    5967 scope.go:122] "RemoveContainer" containerID="5f68d020bd72ba188cde89104a3c042a420627db2f2573b4b5729aa016b3ae7e"
Oct 30 21:01:22.962981 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:22.962935    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" event={"ID":"9c9a8bbd-bb10-464a-a543-6ebc193166a0","Type":"ContainerStarted","Data":"e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"}
Oct 30 21:01:22.963370 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:22.963001    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" event={"ID":"9c9a8bbd-bb10-464a-a543-6ebc193166a0","Type":"ContainerStarted","Data":"3f10c1713c87d291ce66f9f2aea7d71f1f1ab27b9038934ad55c573fce179eee"}
Oct 30 21:01:23.180575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:23.180524    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podStartSLOduration=2.180510295 podStartE2EDuration="2.180510295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:23.12936001 +0000 UTC m=+2463.730161971" watchObservedRunningTime="2026-10-30 21:01:23.180510295 +0000 UTC m=+2463.781312316"
Oct 30 21:01:23.530268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:23.530234    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:23.542029 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:23.542002    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:23.969804 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:23.969772    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:23.970679 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:23.970659    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:01:24.692378 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:24.692353    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"]
Oct 30 21:01:24.741803 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:24.741778    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e2fc1-l7z9j"]
Oct 30 21:01:25.542873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:25.542841    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2cadff-e237-4e97-af50-839f84aa4dee" path="/var/lib/kubelet/pods/9b2cadff-e237-4e97-af50-839f84aa4dee/volumes"
Oct 30 21:01:30.023351 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.023304    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"]
Oct 30 21:01:30.039415 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.039387    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.091448 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.091380    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"]
Oct 30 21:01:30.130454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.130412    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmsj\" (UniqueName: \"kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj\") pod \"payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.130644 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.130621    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.134163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.134140    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2fa881\""
Oct 30 21:01:30.134163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.134141    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2fa881-dockercfg-ghw7k\""
Oct 30 21:01:30.193961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.193924    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"]
Oct 30 21:01:30.199019 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.198989    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.231400 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.231373    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmsj\" (UniqueName: \"kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj\") pod \"payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.231567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.231477    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.232100 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.232083    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.254434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.254405    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"]
Oct 30 21:01:30.287964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.287888    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmsj\" (UniqueName: \"kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj\") pod \"payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.357786 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.343484    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqn5\" (UniqueName: \"kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5\") pod \"payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.357786 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.343567    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.378213 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.378179    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:30.458160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.453170    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqn5\" (UniqueName: \"kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5\") pod \"payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.458160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.453245    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.458160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.454259    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.509663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.509628    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqn5\" (UniqueName: \"kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5\") pod \"payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.517413 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.517387    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:30.761931 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.761903    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"]
Oct 30 21:01:30.763387 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:01:30.763363    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d20da6c_ea9e_4bf8_9e7d_228838469418.slice/crio-6b0bd3b6cbddb376ce089b4aace6e5f12aac64a35e2b9bed8d98d9eca3812483 WatchSource:0}: Error finding container 6b0bd3b6cbddb376ce089b4aace6e5f12aac64a35e2b9bed8d98d9eca3812483: Status 404 returned error can't find the container with id 6b0bd3b6cbddb376ce089b4aace6e5f12aac64a35e2b9bed8d98d9eca3812483
Oct 30 21:01:30.893530 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:30.893505    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"]
Oct 30 21:01:30.894791 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:01:30.894759    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9047596e_8fb5_479b_83c0_2a7b83bfb82d.slice/crio-4556e6d96dfacbda7c2506a4843c86266108f9c5e1a48aed7af882602ce95a56 WatchSource:0}: Error finding container 4556e6d96dfacbda7c2506a4843c86266108f9c5e1a48aed7af882602ce95a56: Status 404 returned error can't find the container with id 4556e6d96dfacbda7c2506a4843c86266108f9c5e1a48aed7af882602ce95a56
Oct 30 21:01:31.015815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.015786    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" event={"ID":"9047596e-8fb5-479b-83c0-2a7b83bfb82d","Type":"ContainerStarted","Data":"e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c"}
Oct 30 21:01:31.015955 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.015836    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" event={"ID":"9047596e-8fb5-479b-83c0-2a7b83bfb82d","Type":"ContainerStarted","Data":"4556e6d96dfacbda7c2506a4843c86266108f9c5e1a48aed7af882602ce95a56"}
Oct 30 21:01:31.015955 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.015858    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:31.017474 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.017452    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6" event={"ID":"6d20da6c-ea9e-4bf8-9e7d-228838469418","Type":"ContainerStarted","Data":"1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d"}
Oct 30 21:01:31.017583 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.017499    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6" event={"ID":"6d20da6c-ea9e-4bf8-9e7d-228838469418","Type":"ContainerStarted","Data":"6b0bd3b6cbddb376ce089b4aace6e5f12aac64a35e2b9bed8d98d9eca3812483"}
Oct 30 21:01:31.017583 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.017519    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:31.051039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.051015    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"]
Oct 30 21:01:31.055126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.055108    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.107944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.107836    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:31.107944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.107853    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:01:31.108142 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.108029    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:01:31.108203 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.108170    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:01:31.111363 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.111346    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"]
Oct 30 21:01:31.131190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.131142    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" podStartSLOduration=1.131126209 podStartE2EDuration="1.131126209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:31.129102834 +0000 UTC m=+2471.729904794" watchObservedRunningTime="2026-10-30 21:01:31.131126209 +0000 UTC m=+2471.731928170"
Oct 30 21:01:31.161696 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.161670    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmpb\" (UniqueName: \"kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.161819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.161801    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.161926 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.161910    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.250196 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.250136    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6" podStartSLOduration=2.2501188389999998 podStartE2EDuration="2.250118839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:31.245260495 +0000 UTC m=+2471.846062456" watchObservedRunningTime="2026-10-30 21:01:31.250118839 +0000 UTC m=+2471.850920800"
Oct 30 21:01:31.263064 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.263034    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.263250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.263089    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.263340 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.263264    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmpb\" (UniqueName: \"kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.264065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.264040    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.265793 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.265771    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.320117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.320083    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmpb\" (UniqueName: \"kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb\") pod \"maas-api-revoke-keys-e2e-api-b-23491e-l2f2v\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.367673 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.367611    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:31.762246 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.762224    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"]
Oct 30 21:01:31.763254 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:01:31.763227    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929e618f_8794_488c_85aa_8b7353ec47cf.slice/crio-662aeb954d657d57a9ce7c7b40a1605f74569ed53ff3e8d5f8d817e52685919b WatchSource:0}: Error finding container 662aeb954d657d57a9ce7c7b40a1605f74569ed53ff3e8d5f8d817e52685919b: Status 404 returned error can't find the container with id 662aeb954d657d57a9ce7c7b40a1605f74569ed53ff3e8d5f8d817e52685919b
Oct 30 21:01:31.972894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.972861    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"]
Oct 30 21:01:31.973126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:31.973100    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy" containerID="cri-o://f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab" gracePeriod=30
Oct 30 21:01:31.982409 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:01:31.982378    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:01:32.028302 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:32.028231    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v" event={"ID":"929e618f-8794-488c-85aa-8b7353ec47cf","Type":"ContainerStarted","Data":"662aeb954d657d57a9ce7c7b40a1605f74569ed53ff3e8d5f8d817e52685919b"}
Oct 30 21:01:33.038493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:33.038461    5967 generic.go:356] "Generic (PLEG): container finished" podID="929e618f-8794-488c-85aa-8b7353ec47cf" containerID="ebd885e43ee9fd595c778f3605de05cb133348284030aa956919a9d6baa8dbe9" exitCode=0
Oct 30 21:01:33.038960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:33.038549    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v" event={"ID":"929e618f-8794-488c-85aa-8b7353ec47cf","Type":"ContainerDied","Data":"ebd885e43ee9fd595c778f3605de05cb133348284030aa956919a9d6baa8dbe9"}
Oct 30 21:01:34.197288 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.197266    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:34.300035 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.299979    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb\" (UniqueName: \"kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb\") pod \"929e618f-8794-488c-85aa-8b7353ec47cf\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") "
Oct 30 21:01:34.300159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.300043    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca\") pod \"929e618f-8794-488c-85aa-8b7353ec47cf\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") "
Oct 30 21:01:34.300159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.300106    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca\") pod \"929e618f-8794-488c-85aa-8b7353ec47cf\" (UID: \"929e618f-8794-488c-85aa-8b7353ec47cf\") "
Oct 30 21:01:34.300481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.300458    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca" pod "929e618f-8794-488c-85aa-8b7353ec47cf" (UID: "929e618f-8794-488c-85aa-8b7353ec47cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:34.300575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.300561    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:34.302291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.302267    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca" pod "929e618f-8794-488c-85aa-8b7353ec47cf" (UID: "929e618f-8794-488c-85aa-8b7353ec47cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:34.302368 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.302276    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb" pod "929e618f-8794-488c-85aa-8b7353ec47cf" (UID: "929e618f-8794-488c-85aa-8b7353ec47cf"). InnerVolumeSpecName "kube-api-access-kvmpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:34.401469 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.401437    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmpb\" (UniqueName: \"kubernetes.io/projected/929e618f-8794-488c-85aa-8b7353ec47cf-kube-api-access-kvmpb\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:34.401469 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:34.401467    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929e618f-8794-488c-85aa-8b7353ec47cf-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.055687 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:35.055640    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v" event={"ID":"929e618f-8794-488c-85aa-8b7353ec47cf","Type":"ContainerDied","Data":"662aeb954d657d57a9ce7c7b40a1605f74569ed53ff3e8d5f8d817e52685919b"}
Oct 30 21:01:35.055882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:35.055706    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="662aeb954d657d57a9ce7c7b40a1605f74569ed53ff3e8d5f8d817e52685919b"
Oct 30 21:01:35.055882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:35.055664    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"
Oct 30 21:01:37.238444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.238420    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 21:01:37.339510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339451    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.339510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339498    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.339813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339591    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.339813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339693    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.339813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339721    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:37.339813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339767    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.339813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339804    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.340090 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339855    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.340090 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339891    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.340090 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339935    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo\") pod \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\" (UID: \"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7\") "
Oct 30 21:01:37.340090 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.339954    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:37.340090 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340056    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:37.340323 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340091    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:37.340323 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340169    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:37.340446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340431    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.340498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340450    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.340498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340462    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.340498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340476    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.340498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.340487    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.342446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.342423    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:37.342446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.342434    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:37.342646 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.342631    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:01:37.342906 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.342886    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq" pod "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" (UID: "2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7"). InnerVolumeSpecName "kube-api-access-pkjmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:37.441797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.441762    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjmq\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-kube-api-access-pkjmq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.441797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.441798    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.441996 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.441811    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:37.441996 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:37.441824    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:38.077189 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.077151    5967 generic.go:356] "Generic (PLEG): container finished" podID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerID="f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab" exitCode=0
Oct 30 21:01:38.077377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.077215    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"
Oct 30 21:01:38.077377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.077236    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" event={"ID":"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7","Type":"ContainerDied","Data":"f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab"}
Oct 30 21:01:38.077377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.077329    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg" event={"ID":"2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7","Type":"ContainerDied","Data":"07bee2be673316674128476d9c1d8800fb6f8fb6a9a2ebef3b08a614489005d5"}
Oct 30 21:01:38.077377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.077358    5967 scope.go:122] "RemoveContainer" containerID="f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab"
Oct 30 21:01:38.087294 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.087279    5967 scope.go:122] "RemoveContainer" containerID="ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"
Oct 30 21:01:38.096199 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.096044    5967 scope.go:122] "RemoveContainer" containerID="f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab"
Oct 30 21:01:38.096324 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:01:38.096306    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab\": container with ID starting with f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab not found: ID does not exist" containerID="f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab"
Oct 30 21:01:38.096370 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.096331    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab"} err="failed to get container status \"f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab\": rpc error: code = NotFound desc = could not find container \"f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab\": container with ID starting with f1bc1deb77b953928a502e56f4e2107461697c6e794fb2e014d9b6ea496e89ab not found: ID does not exist"
Oct 30 21:01:38.096370 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.096347    5967 scope.go:122] "RemoveContainer" containerID="ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"
Oct 30 21:01:38.096569 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:01:38.096552    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae\": container with ID starting with ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae not found: ID does not exist" containerID="ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"
Oct 30 21:01:38.096610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.096577    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae"} err="failed to get container status \"ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae\": rpc error: code = NotFound desc = could not find container \"ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae\": container with ID starting with ee32d48faf2947031b542340a85c73b421663715168eac280c24e50f6c7747ae not found: ID does not exist"
Oct 30 21:01:38.289946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.289912    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"]
Oct 30 21:01:38.339479 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:38.339420    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg"]
Oct 30 21:01:39.542483 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:39.542451    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" path="/var/lib/kubelet/pods/2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7/volumes"
Oct 30 21:01:42.031483 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:42.031405    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:01:42.031965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:42.031493    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:01:59.546842 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:59.546808    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"]
Oct 30 21:01:59.549632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:01:59.549605    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd2d96-mlpsd"]
Oct 30 21:02:01.543984 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:01.543949    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a696167-db92-4f40-b647-a1480edec193" path="/var/lib/kubelet/pods/3a696167-db92-4f40-b647-a1480edec193/volumes"
Oct 30 21:02:01.980818 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:02:01.980783    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:02:06.886480 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.886446    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"]
Oct 30 21:02:06.887119 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887104    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy"
Oct 30 21:02:06.887169 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887122    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy"
Oct 30 21:02:06.887219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887186    5967 cpu_manager.go:414] "containerMap: removing container" podUID="929e618f-8794-488c-85aa-8b7353ec47cf" containerName="revoke-keys"
Oct 30 21:02:06.887219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887195    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929e618f-8794-488c-85aa-8b7353ec47cf" containerName="revoke-keys"
Oct 30 21:02:06.887219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887217    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy"
Oct 30 21:02:06.887320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887228    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy"
Oct 30 21:02:06.887359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887329    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="929e618f-8794-488c-85aa-8b7353ec47cf" containerName="revoke-keys"
Oct 30 21:02:06.887359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887337    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy"
Oct 30 21:02:06.887359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.887344    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc9a219-f3e6-4bd8-8cb8-0856316cf0d7" containerName="istio-proxy"
Oct 30 21:02:06.892154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.892132    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:06.928538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.928504    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9s9q\" (UniqueName: \"kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:06.928732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.928570    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:06.928732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.928728    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:06.948266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.948227    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"]
Oct 30 21:02:06.993677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.993596    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:06.993677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.993611    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:02:06.993677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.993628    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:02:06.993677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:06.993674    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:02:07.030414 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.030370    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.030598 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.030537    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9s9q\" (UniqueName: \"kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.030598 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.030586    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.031302 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.031282    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.032901 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.032871    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.087562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.087528    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9s9q\" (UniqueName: \"kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q\") pod \"maas-api-revoke-keys-e2e-api-b-92f229-bqt9w\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.206655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.206551    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:07.587197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:07.587165    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"]
Oct 30 21:02:07.589019 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:02:07.588984    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcc73fd_4a81_4700_96c2_469f0d1bd5bf.slice/crio-26b18fe887fd79be5187b10083d75ac4c65323a061ea803a72500f699eed45ae WatchSource:0}: Error finding container 26b18fe887fd79be5187b10083d75ac4c65323a061ea803a72500f699eed45ae: Status 404 returned error can't find the container with id 26b18fe887fd79be5187b10083d75ac4c65323a061ea803a72500f699eed45ae
Oct 30 21:02:08.286124 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:08.286085    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w" event={"ID":"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf","Type":"ContainerStarted","Data":"33b713c3a03ca8732a3813d25e4a5078f4c02f0b2058931c1d336b86c2b6fe6e"}
Oct 30 21:02:08.286879 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:08.286141    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w" event={"ID":"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf","Type":"ContainerStarted","Data":"26b18fe887fd79be5187b10083d75ac4c65323a061ea803a72500f699eed45ae"}
Oct 30 21:02:08.402075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:08.402022    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w" podStartSLOduration=2.402006322 podStartE2EDuration="2.402006322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:08.400090545 +0000 UTC m=+2509.000892506" watchObservedRunningTime="2026-10-30 21:02:08.402006322 +0000 UTC m=+2509.002808293"
Oct 30 21:02:09.295621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:09.295584    5967 generic.go:356] "Generic (PLEG): container finished" podID="5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" containerID="33b713c3a03ca8732a3813d25e4a5078f4c02f0b2058931c1d336b86c2b6fe6e" exitCode=0
Oct 30 21:02:09.296060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:09.295680    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w" event={"ID":"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf","Type":"ContainerDied","Data":"33b713c3a03ca8732a3813d25e4a5078f4c02f0b2058931c1d336b86c2b6fe6e"}
Oct 30 21:02:10.454424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.454396    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:10.575604 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.575525    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca\") pod \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") "
Oct 30 21:02:10.575812 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.575687    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q\" (UniqueName: \"kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q\") pod \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") "
Oct 30 21:02:10.575812 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.575725    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca\") pod \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\" (UID: \"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf\") "
Oct 30 21:02:10.576190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.576162    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca" pod "5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" (UID: "5fcc73fd-4a81-4700-96c2-469f0d1bd5bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:10.578308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.578282    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca" pod "5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" (UID: "5fcc73fd-4a81-4700-96c2-469f0d1bd5bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:10.578902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.578877    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q" pod "5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" (UID: "5fcc73fd-4a81-4700-96c2-469f0d1bd5bf"). InnerVolumeSpecName "kube-api-access-z9s9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:10.677039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.677000    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9s9q\" (UniqueName: \"kubernetes.io/projected/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-kube-api-access-z9s9q\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.677039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.677037    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.677312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:10.677053    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.310239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:11.310208    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"
Oct 30 21:02:11.310431 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:11.310243    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w" event={"ID":"5fcc73fd-4a81-4700-96c2-469f0d1bd5bf","Type":"ContainerDied","Data":"26b18fe887fd79be5187b10083d75ac4c65323a061ea803a72500f699eed45ae"}
Oct 30 21:02:11.310431 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:11.310291    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b18fe887fd79be5187b10083d75ac4c65323a061ea803a72500f699eed45ae"
Oct 30 21:02:22.049486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:22.049449    5967 scope.go:122] "RemoveContainer" containerID="6dfe993e411a188926edaa523ab064580336b2a02186b362df16595797ed9e9d"
Oct 30 21:02:32.002790 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:02:32.002675    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:02:34.983849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:34.983816    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"]
Oct 30 21:02:34.984469 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:34.984059    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy" containerID="cri-o://62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd" gracePeriod=30
Oct 30 21:02:36.488235 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:36.488203    5967 generic.go:356] "Generic (PLEG): container finished" podID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerID="e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4" exitCode=0
Oct 30 21:02:36.488728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:36.488282    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" event={"ID":"9c9a8bbd-bb10-464a-a543-6ebc193166a0","Type":"ContainerDied","Data":"e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"}
Oct 30 21:02:36.488728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:36.488713    5967 scope.go:122] "RemoveContainer" containerID="e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"
Oct 30 21:02:36.490229 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:36.490197    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:02:36.490332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:36.490271    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:02:36.490332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:36.490308    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:02:37.502417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:37.502383    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" event={"ID":"9c9a8bbd-bb10-464a-a543-6ebc193166a0","Type":"ContainerStarted","Data":"48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c"}
Oct 30 21:02:37.530120 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:37.530080    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:02:37.531593 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:37.531565    5967 patch_prober.go:29] interesting pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:02:37.531759 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:37.531616    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 30 21:02:38.531336 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:38.531299    5967 patch_prober.go:29] interesting pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.531792 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:38.531356    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 30 21:02:38.972950 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:38.972852    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:02:39.046128 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.046080    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"]
Oct 30 21:02:39.101123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.101081    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"]
Oct 30 21:02:39.103731 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.103698    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a748b0-f6qkt"]
Oct 30 21:02:39.106195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.106166    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6d1a0f-nb7qc"]
Oct 30 21:02:39.530927 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.530894    5967 patch_prober.go:29] interesting pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:02:39.531154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.530971    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 30 21:02:39.542655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.542627    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a104cceb-0668-438b-a90f-b64b75162f68" path="/var/lib/kubelet/pods/a104cceb-0668-438b-a90f-b64b75162f68/volumes"
Oct 30 21:02:39.543114 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:39.543099    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad63ea7-b252-470b-9ab8-65323de7bdd2" path="/var/lib/kubelet/pods/fad63ea7-b252-470b-9ab8-65323de7bdd2/volumes"
Oct 30 21:02:40.297255 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.296964    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 21:02:40.363369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363330    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.363571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363392    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.363571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363468    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.363571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363511    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.363802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363584    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.363802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363648    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.363802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363656    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:40.363802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363679    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.364042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363940    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:40.364042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.363950    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.364042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364006    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v\") pod \"68229717-55df-40ff-9c71-1aa4daa43ae8\" (UID: \"68229717-55df-40ff-9c71-1aa4daa43ae8\") "
Oct 30 21:02:40.364042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364007    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:40.364042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364021    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:40.364277 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364090    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:40.364730 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364703    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.364875 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364734    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.364875 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364786    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.364875 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364800    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68229717-55df-40ff-9c71-1aa4daa43ae8-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.364875 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.364814    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.365978 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.365954    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:40.366104 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.366086    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:40.366303 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.366283    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "kube-api-access-vv22v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:40.366383 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.366370    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token" pod "68229717-55df-40ff-9c71-1aa4daa43ae8" (UID: "68229717-55df-40ff-9c71-1aa4daa43ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:40.465733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.465699    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.465733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.465731    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv22v\" (UniqueName: \"kubernetes.io/projected/68229717-55df-40ff-9c71-1aa4daa43ae8-kube-api-access-vv22v\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.465733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.465772    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.466065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.465792    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68229717-55df-40ff-9c71-1aa4daa43ae8-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.524217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.524127    5967 generic.go:356] "Generic (PLEG): container finished" podID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerID="62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd" exitCode=0
Oct 30 21:02:40.524217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.524171    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" event={"ID":"68229717-55df-40ff-9c71-1aa4daa43ae8","Type":"ContainerDied","Data":"62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd"}
Oct 30 21:02:40.524217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.524181    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"
Oct 30 21:02:40.524217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.524213    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc" event={"ID":"68229717-55df-40ff-9c71-1aa4daa43ae8","Type":"ContainerDied","Data":"0a1147b973d95d06ec6548df55f836f3f6258c1b77ca2047b59fff403b4eda35"}
Oct 30 21:02:40.524567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.524235    5967 scope.go:122] "RemoveContainer" containerID="62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd"
Oct 30 21:02:40.530439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.530408    5967 patch_prober.go:29] interesting pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.530596 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.530468    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 30 21:02:40.536334 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.536309    5967 scope.go:122] "RemoveContainer" containerID="62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd"
Oct 30 21:02:40.536700 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:02:40.536668    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd\": container with ID starting with 62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd not found: ID does not exist" containerID="62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd"
Oct 30 21:02:40.536810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.536710    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd"} err="failed to get container status \"62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd\": rpc error: code = NotFound desc = could not find container \"62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd\": container with ID starting with 62bc9b6d45aaf9bdc6668902370c479ba6783c355abd743220f03ecef72511dd not found: ID does not exist"
Oct 30 21:02:40.698240 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.698208    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"]
Oct 30 21:02:40.700851 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:40.700829    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc"]
Oct 30 21:02:41.531053 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:41.531021    5967 patch_prober.go:29] interesting pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:02:41.531279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:41.531092    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 30 21:02:41.544354 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:41.544315    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" path="/var/lib/kubelet/pods/68229717-55df-40ff-9c71-1aa4daa43ae8/volumes"
Oct 30 21:02:42.530525 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:42.530482    5967 patch_prober.go:29] interesting pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:02:42.531091 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:42.530560    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 30 21:02:43.544163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:43.544133    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:02:43.545706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:43.545684    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:02:53.896034 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.895989    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"]
Oct 30 21:02:53.896909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.896889    5967 cpu_manager.go:414] "containerMap: removing container" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy"
Oct 30 21:02:53.897036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.896912    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy"
Oct 30 21:02:53.897036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.896939    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" containerName="revoke-keys"
Oct 30 21:02:53.897036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.896946    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" containerName="revoke-keys"
Oct 30 21:02:53.897205 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.897107    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="68229717-55df-40ff-9c71-1aa4daa43ae8" containerName="istio-proxy"
Oct 30 21:02:53.897205 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.897122    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" containerName="revoke-keys"
Oct 30 21:02:53.900612 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.900586    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949333 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949298    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949333 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949337    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949605 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949496    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949605 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949573    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949635    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949697    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949842 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949729    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949842 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949767    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.949931 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.949866    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwv2\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:53.972672 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:53.972632    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"]
Oct 30 21:02:54.004031 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.003995    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a064b4-openshift-default-dockercfg-mpxmn\""
Oct 30 21:02:54.052588 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052535    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.052886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052627    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwv2\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.052886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052765    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.052886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052797    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.052886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052852    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.053121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052915    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.053121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.052947    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.053121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.053001    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.053121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.053050    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.053957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.053910    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.056103 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.055082    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.056292 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.055557    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.059239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.055794    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.059574 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.056037    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.061512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.061478    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.065980 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.065953    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.110846 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.110807    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.110846 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.110835    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwv2\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2\") pod \"e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") " pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.215055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.215025    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:54.665593 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.665486    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"]
Oct 30 21:02:54.692040 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:02:54.692010    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8128db7f_338a_4298_9ae2_92f650cbc9d6.slice/crio-6df32ebe0c0ca1a8fc78ef52aa0d70ec2e6a1fc6ec4d025ccf1248112a26ca6f WatchSource:0}: Error finding container 6df32ebe0c0ca1a8fc78ef52aa0d70ec2e6a1fc6ec4d025ccf1248112a26ca6f: Status 404 returned error can't find the container with id 6df32ebe0c0ca1a8fc78ef52aa0d70ec2e6a1fc6ec4d025ccf1248112a26ca6f
Oct 30 21:02:54.694559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.694521    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:02:54.694655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.694603    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:02:54.694655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:54.694644    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:02:55.640808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:55.640765    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" event={"ID":"8128db7f-338a-4298-9ae2-92f650cbc9d6","Type":"ContainerStarted","Data":"bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"}
Oct 30 21:02:55.641348 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:55.640837    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" event={"ID":"8128db7f-338a-4298-9ae2-92f650cbc9d6","Type":"ContainerStarted","Data":"6df32ebe0c0ca1a8fc78ef52aa0d70ec2e6a1fc6ec4d025ccf1248112a26ca6f"}
Oct 30 21:02:55.809518 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:55.809463    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podStartSLOduration=2.809446308 podStartE2EDuration="2.809446308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:55.807865768 +0000 UTC m=+2556.408667732" watchObservedRunningTime="2026-10-30 21:02:55.809446308 +0000 UTC m=+2556.410248268"
Oct 30 21:02:56.215712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:56.215674    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:56.221901 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:56.221871    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:56.647667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:56.647582    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:02:56.648700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:02:56.648683    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:03:02.012696 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:02.012658    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:03:08.283704 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:08.283664    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"]
Oct 30 21:03:08.332653 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:08.332616    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-23491e-l2f2v"]
Oct 30 21:03:09.543718 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:09.543683    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929e618f-8794-488c-85aa-8b7353ec47cf" path="/var/lib/kubelet/pods/929e618f-8794-488c-85aa-8b7353ec47cf/volumes"
Oct 30 21:03:14.951174 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:14.951139    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"]
Oct 30 21:03:14.951689 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:14.951335    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy" containerID="cri-o://2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd" gracePeriod=30
Oct 30 21:03:15.956973 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:15.956931    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"]
Oct 30 21:03:15.961203 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:15.961178    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.017969 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.017935    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"]
Oct 30 21:03:16.062719 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.062668    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.062945 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.062722    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm5n\" (UniqueName: \"kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.063025 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.062937    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.063609 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.063582    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:16.063776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.063665    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:16.063776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.063707    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:16.063776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.063707    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:03:16.164193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.164156    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.164193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.164195    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm5n\" (UniqueName: \"kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.164421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.164293    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.164858 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.164841    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.167140 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.167121    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.221495 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.221457    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm5n\" (UniqueName: \"kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n\") pod \"maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.275271 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.275231    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:16.650781 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.650752    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"]
Oct 30 21:03:16.652845 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:03:16.652805    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40b216f_90f8_42a0_9e62_d4c0733ccf8c.slice/crio-0ec27fc0da7b950c3996dfefabdbfb9d8445c429b084d6f91bf141add24da4a0 WatchSource:0}: Error finding container 0ec27fc0da7b950c3996dfefabdbfb9d8445c429b084d6f91bf141add24da4a0: Status 404 returned error can't find the container with id 0ec27fc0da7b950c3996dfefabdbfb9d8445c429b084d6f91bf141add24da4a0
Oct 30 21:03:16.799848 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:16.799810    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj" event={"ID":"d40b216f-90f8-42a0-9e62-d4c0733ccf8c","Type":"ContainerStarted","Data":"0ec27fc0da7b950c3996dfefabdbfb9d8445c429b084d6f91bf141add24da4a0"}
Oct 30 21:03:17.808414 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:17.808319    5967 generic.go:356] "Generic (PLEG): container finished" podID="d40b216f-90f8-42a0-9e62-d4c0733ccf8c" containerID="7b161a46b4ebf36659f44b12504bf019686df86e53787dd9c6fb16fffcddde97" exitCode=0
Oct 30 21:03:17.808858 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:17.808410    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj" event={"ID":"d40b216f-90f8-42a0-9e62-d4c0733ccf8c","Type":"ContainerDied","Data":"7b161a46b4ebf36659f44b12504bf019686df86e53787dd9c6fb16fffcddde97"}
Oct 30 21:03:18.968066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:18.968039    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:19.104183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.104096    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n\" (UniqueName: \"kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n\") pod \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") "
Oct 30 21:03:19.104369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.104233    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca\") pod \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") "
Oct 30 21:03:19.104438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.104384    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca\") pod \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\" (UID: \"d40b216f-90f8-42a0-9e62-d4c0733ccf8c\") "
Oct 30 21:03:19.104840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.104808    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca" pod "d40b216f-90f8-42a0-9e62-d4c0733ccf8c" (UID: "d40b216f-90f8-42a0-9e62-d4c0733ccf8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:19.104983 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.104930    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:19.106432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.106398    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n" pod "d40b216f-90f8-42a0-9e62-d4c0733ccf8c" (UID: "d40b216f-90f8-42a0-9e62-d4c0733ccf8c"). InnerVolumeSpecName "kube-api-access-xxm5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:19.106543 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.106496    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca" pod "d40b216f-90f8-42a0-9e62-d4c0733ccf8c" (UID: "d40b216f-90f8-42a0-9e62-d4c0733ccf8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:19.206025 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.205991    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:19.206025 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.206019    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxm5n\" (UniqueName: \"kubernetes.io/projected/d40b216f-90f8-42a0-9e62-d4c0733ccf8c-kube-api-access-xxm5n\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:19.825123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.825035    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj" event={"ID":"d40b216f-90f8-42a0-9e62-d4c0733ccf8c","Type":"ContainerDied","Data":"0ec27fc0da7b950c3996dfefabdbfb9d8445c429b084d6f91bf141add24da4a0"}
Oct 30 21:03:19.825123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.825102    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ec27fc0da7b950c3996dfefabdbfb9d8445c429b084d6f91bf141add24da4a0"
Oct 30 21:03:19.825123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:19.825046    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"
Oct 30 21:03:20.217977 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.217952    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 21:03:20.319946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.319913    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.319946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.319952    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.319971    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320034    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320087    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320121    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320245    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320329    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.320450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320338    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320431    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.320450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320433    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt\") pod \"0d7dfda0-0021-4b45-bc15-044422fad237\" (UID: \"0d7dfda0-0021-4b45-bc15-044422fad237\") "
Oct 30 21:03:20.320681 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320451    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:20.321379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.320959    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.321379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.321308    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.321379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.321327    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7dfda0-0021-4b45-bc15-044422fad237-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.321379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.321342    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.321379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.321357    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.321793 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.321772    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.322519 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.322496    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.323382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.323360    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.323464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.323427    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:20.323464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.323447    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt" pod "0d7dfda0-0021-4b45-bc15-044422fad237" (UID: "0d7dfda0-0021-4b45-bc15-044422fad237"). InnerVolumeSpecName "kube-api-access-r4fwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.419061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.418975    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"]
Oct 30 21:03:20.422690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.422580    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.422690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.422620    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.422690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.422636    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7dfda0-0021-4b45-bc15-044422fad237-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.422690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.422650    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4fwt\" (UniqueName: \"kubernetes.io/projected/0d7dfda0-0021-4b45-bc15-044422fad237-kube-api-access-r4fwt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.422690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.422665    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7dfda0-0021-4b45-bc15-044422fad237-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.833399 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.833364    5967 generic.go:356] "Generic (PLEG): container finished" podID="0d7dfda0-0021-4b45-bc15-044422fad237" containerID="2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd" exitCode=0
Oct 30 21:03:20.833816 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.833789    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" event={"ID":"0d7dfda0-0021-4b45-bc15-044422fad237","Type":"ContainerDied","Data":"2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd"}
Oct 30 21:03:20.834013 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.833992    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp" event={"ID":"0d7dfda0-0021-4b45-bc15-044422fad237","Type":"ContainerDied","Data":"999dfe2fc38fa48404e19c83a6d5a3ffca81cca27af2a72d802fe65fd2e2246b"}
Oct 30 21:03:20.834133 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.833889    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"
Oct 30 21:03:20.834133 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.834028    5967 scope.go:122] "RemoveContainer" containerID="2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd"
Oct 30 21:03:20.846264 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.846247    5967 scope.go:122] "RemoveContainer" containerID="997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"
Oct 30 21:03:20.857119 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.857093    5967 scope.go:122] "RemoveContainer" containerID="2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd"
Oct 30 21:03:20.857546 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:20.857522    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd\": container with ID starting with 2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd not found: ID does not exist" containerID="2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd"
Oct 30 21:03:20.857618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.857555    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd"} err="failed to get container status \"2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd\": rpc error: code = NotFound desc = could not find container \"2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd\": container with ID starting with 2d0f8132b56e8020cffd597055192376472c66a23322b772b66164048cc1afbd not found: ID does not exist"
Oct 30 21:03:20.857618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.857574    5967 scope.go:122] "RemoveContainer" containerID="997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"
Oct 30 21:03:20.857864 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:20.857843    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d\": container with ID starting with 997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d not found: ID does not exist" containerID="997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"
Oct 30 21:03:20.857923 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.857872    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d"} err="failed to get container status \"997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d\": rpc error: code = NotFound desc = could not find container \"997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d\": container with ID starting with 997590a004111d0eb1cafe64d5354ccb16939bbc661406bc414f5b656ce7613d not found: ID does not exist"
Oct 30 21:03:20.996681 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:20.996640    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"]
Oct 30 21:03:21.045588 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:21.045555    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp"]
Oct 30 21:03:21.543544 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:21.543506    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" path="/var/lib/kubelet/pods/0d7dfda0-0021-4b45-bc15-044422fad237/volumes"
Oct 30 21:03:22.086574 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:22.086534    5967 scope.go:122] "RemoveContainer" containerID="1d8eb62243b9e22b3e3c8507cce6e3c03d96e87929a3605f7ff6b6e6d27b2a7c"
Oct 30 21:03:22.096794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:22.096767    5967 scope.go:122] "RemoveContainer" containerID="e4fa8b1010736509cbd70a7bc8e9108a818f48ad85d2e9a8c1d5727e5a009c96"
Oct 30 21:03:22.108374 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:22.108354    5967 scope.go:122] "RemoveContainer" containerID="70cd0b81e624e080e2816bfc516259b8c9b890e0a17dce5ab4819eb0b5f8e019"
Oct 30 21:03:23.046060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:23.046023    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"]
Oct 30 21:03:23.096215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:23.096180    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"]
Oct 30 21:03:23.098525 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:23.098511    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8549e3cb-zf8rw"]
Oct 30 21:03:23.100717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:23.100691    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf3e76e-fdmf4"]
Oct 30 21:03:23.550086 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:23.550054    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d296c8-49bc-4d0c-885f-d781dc5ae26b" path="/var/lib/kubelet/pods/65d296c8-49bc-4d0c-885f-d781dc5ae26b/volumes"
Oct 30 21:03:23.550493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:23.550480    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f859df-b5e7-4821-bb2a-7409463fcdcc" path="/var/lib/kubelet/pods/f4f859df-b5e7-4821-bb2a-7409463fcdcc/volumes"
Oct 30 21:03:27.450642 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:27.450607    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"]
Oct 30 21:03:27.710343 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:27.710253    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"]
Oct 30 21:03:29.086898 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.086859    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"]
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.088283    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d40b216f-90f8-42a0-9e62-d4c0733ccf8c" containerName="revoke-keys"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.091400    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40b216f-90f8-42a0-9e62-d4c0733ccf8c" containerName="revoke-keys"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.091459    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.091471    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.091780    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.091806    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.091824    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40b216f-90f8-42a0-9e62-d4c0733ccf8c" containerName="revoke-keys"
Oct 30 21:03:29.100788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.097864    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.123727 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.123697    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"]
Oct 30 21:03:29.149901 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.149868    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:29.150084 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.149925    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:29.150084 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.149911    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:29.169795 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.169734    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:03:29.232735 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.232696    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.232935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.232911    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.232985 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.232939    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scf44\" (UniqueName: \"kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.334773 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.334728    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.334965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.334852    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.334965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.334873    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scf44\" (UniqueName: \"kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.335594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.335502    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.337414 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.337361    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.392347 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.392311    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scf44\" (UniqueName: \"kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44\") pod \"maas-api-revoke-sub-c87e3393-kj5qk\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.414759 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.414708    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:29.792314 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.792283    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"]
Oct 30 21:03:29.794086 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:03:29.794054    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02ebacf_ac9c_4d7e_841e_81d36d8631d1.slice/crio-cc0057bdd1f604b82a851beb22ad30c9a1ba46f52128bdd374e27ef0d450f6fe WatchSource:0}: Error finding container cc0057bdd1f604b82a851beb22ad30c9a1ba46f52128bdd374e27ef0d450f6fe: Status 404 returned error can't find the container with id cc0057bdd1f604b82a851beb22ad30c9a1ba46f52128bdd374e27ef0d450f6fe
Oct 30 21:03:29.895245 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:29.895207    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk" event={"ID":"f02ebacf-ac9c-4d7e-841e-81d36d8631d1","Type":"ContainerStarted","Data":"cc0057bdd1f604b82a851beb22ad30c9a1ba46f52128bdd374e27ef0d450f6fe"}
Oct 30 21:03:30.903562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:30.903530    5967 generic.go:356] "Generic (PLEG): container finished" podID="f02ebacf-ac9c-4d7e-841e-81d36d8631d1" containerID="84dbf435f4bac528af696d0ed753171f699c302694bd988bf4c0b8a7c6724a7a" exitCode=0
Oct 30 21:03:30.904038 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:30.903621    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk" event={"ID":"f02ebacf-ac9c-4d7e-841e-81d36d8631d1","Type":"ContainerDied","Data":"84dbf435f4bac528af696d0ed753171f699c302694bd988bf4c0b8a7c6724a7a"}
Oct 30 21:03:32.006560 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:32.006521    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:03:32.099068 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.099042    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:32.282959 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.282922    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca\") pod \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") "
Oct 30 21:03:32.283181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.282977    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca\") pod \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") "
Oct 30 21:03:32.283181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.283143    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44\" (UniqueName: \"kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44\") pod \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\" (UID: \"f02ebacf-ac9c-4d7e-841e-81d36d8631d1\") "
Oct 30 21:03:32.283376 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.283352    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca" pod "f02ebacf-ac9c-4d7e-841e-81d36d8631d1" (UID: "f02ebacf-ac9c-4d7e-841e-81d36d8631d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:32.283628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.283610    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:32.285283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.285263    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca" pod "f02ebacf-ac9c-4d7e-841e-81d36d8631d1" (UID: "f02ebacf-ac9c-4d7e-841e-81d36d8631d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:32.285356 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.285305    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44" pod "f02ebacf-ac9c-4d7e-841e-81d36d8631d1" (UID: "f02ebacf-ac9c-4d7e-841e-81d36d8631d1"). InnerVolumeSpecName "kube-api-access-scf44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:32.384821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.384793    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scf44\" (UniqueName: \"kubernetes.io/projected/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-kube-api-access-scf44\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:32.384821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.384820    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02ebacf-ac9c-4d7e-841e-81d36d8631d1-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:32.451128 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.451091    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" containerName="payload-processing" containerID="cri-o://81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae" gracePeriod=25
Oct 30 21:03:32.710941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.710849    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerName="payload-pre-processing" containerID="cri-o://ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e" gracePeriod=25
Oct 30 21:03:32.918322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.918293    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"
Oct 30 21:03:32.918499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.918294    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk" event={"ID":"f02ebacf-ac9c-4d7e-841e-81d36d8631d1","Type":"ContainerDied","Data":"cc0057bdd1f604b82a851beb22ad30c9a1ba46f52128bdd374e27ef0d450f6fe"}
Oct 30 21:03:32.918499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:32.918422    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc0057bdd1f604b82a851beb22ad30c9a1ba46f52128bdd374e27ef0d450f6fe"
Oct 30 21:03:33.731191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:33.731157    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"]
Oct 30 21:03:33.780531 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:33.780497    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92f229-bqt9w"]
Oct 30 21:03:35.444165 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.444117    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main" containerID="cri-o://cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e" gracePeriod=45
Oct 30 21:03:35.531986 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.531950    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:35.532137 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.531955    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:35.542632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.542604    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcc73fd-4a81-4700-96c2-469f0d1bd5bf" path="/var/lib/kubelet/pods/5fcc73fd-4a81-4700-96c2-469f0d1bd5bf/volumes"
Oct 30 21:03:35.705349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.705285    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 21:03:35.722758 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722720    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home\") pod \"d3d4df14-1364-4b11-8e9e-5adb5289b607\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") "
Oct 30 21:03:35.722902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722813    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs\") pod \"d3d4df14-1364-4b11-8e9e-5adb5289b607\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") "
Oct 30 21:03:35.722947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722901    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache\") pod \"d3d4df14-1364-4b11-8e9e-5adb5289b607\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") "
Oct 30 21:03:35.722947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722931    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5\" (UniqueName: \"kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5\") pod \"d3d4df14-1364-4b11-8e9e-5adb5289b607\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") "
Oct 30 21:03:35.723052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722962    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm\") pod \"d3d4df14-1364-4b11-8e9e-5adb5289b607\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") "
Oct 30 21:03:35.723052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722986    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home" pod "d3d4df14-1364-4b11-8e9e-5adb5289b607" (UID: "d3d4df14-1364-4b11-8e9e-5adb5289b607"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.723052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.722994    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir\") pod \"d3d4df14-1364-4b11-8e9e-5adb5289b607\" (UID: \"d3d4df14-1364-4b11-8e9e-5adb5289b607\") "
Oct 30 21:03:35.723212 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.723154    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache" pod "d3d4df14-1364-4b11-8e9e-5adb5289b607" (UID: "d3d4df14-1364-4b11-8e9e-5adb5289b607"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.723212 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.723201    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir" pod "d3d4df14-1364-4b11-8e9e-5adb5289b607" (UID: "d3d4df14-1364-4b11-8e9e-5adb5289b607"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.723624 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.723604    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.723784 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.723627    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.723784 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.723640    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.725301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.725276    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm" pod "d3d4df14-1364-4b11-8e9e-5adb5289b607" (UID: "d3d4df14-1364-4b11-8e9e-5adb5289b607"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.725301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.725292    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5" pod "d3d4df14-1364-4b11-8e9e-5adb5289b607" (UID: "d3d4df14-1364-4b11-8e9e-5adb5289b607"). InnerVolumeSpecName "kube-api-access-cqfl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:35.725478 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.725346    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs" pod "d3d4df14-1364-4b11-8e9e-5adb5289b607" (UID: "d3d4df14-1364-4b11-8e9e-5adb5289b607"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:35.824584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.824557    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4df14-1364-4b11-8e9e-5adb5289b607-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.824584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.824582    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfl5\" (UniqueName: \"kubernetes.io/projected/d3d4df14-1364-4b11-8e9e-5adb5289b607-kube-api-access-cqfl5\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.824584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.824592    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4df14-1364-4b11-8e9e-5adb5289b607-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.938565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.938534    5967 generic.go:356] "Generic (PLEG): container finished" podID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerID="cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e" exitCode=0
Oct 30 21:03:35.938775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.938573    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" event={"ID":"d3d4df14-1364-4b11-8e9e-5adb5289b607","Type":"ContainerDied","Data":"cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e"}
Oct 30 21:03:35.938775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.938582    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"
Oct 30 21:03:35.938775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.938605    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w" event={"ID":"d3d4df14-1364-4b11-8e9e-5adb5289b607","Type":"ContainerDied","Data":"c23cd7542b3f8d3182313b08c3dd9d0863c5f53e7331137b3f0941e035445e0b"}
Oct 30 21:03:35.938775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.938621    5967 scope.go:122] "RemoveContainer" containerID="cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e"
Oct 30 21:03:35.950906 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.950888    5967 scope.go:122] "RemoveContainer" containerID="cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e"
Oct 30 21:03:35.951196 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:35.951180    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e\": container with ID starting with cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e not found: ID does not exist" containerID="cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e"
Oct 30 21:03:35.951237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:35.951205    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e"} err="failed to get container status \"cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e\": rpc error: code = NotFound desc = could not find container \"cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e\": container with ID starting with cfcc008c155e770a5bfebc2170dbf6c2df1b917bb97563fc6ad15ffaf6f4114e not found: ID does not exist"
Oct 30 21:03:36.103623 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:36.103590    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"]
Oct 30 21:03:36.105854 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:36.105831    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/distinct-2-w2-df031b-kserve-7695cdf7d4-tp59w"]
Oct 30 21:03:37.543245 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:37.543211    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" path="/var/lib/kubelet/pods/d3d4df14-1364-4b11-8e9e-5adb5289b607/volumes"
Oct 30 21:03:41.492302 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.492265    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"]
Oct 30 21:03:41.492989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.492973    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f02ebacf-ac9c-4d7e-841e-81d36d8631d1" containerName="revoke-keys"
Oct 30 21:03:41.493110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.492990    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02ebacf-ac9c-4d7e-841e-81d36d8631d1" containerName="revoke-keys"
Oct 30 21:03:41.493110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.493033    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy"
Oct 30 21:03:41.493110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.493041    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7dfda0-0021-4b45-bc15-044422fad237" containerName="istio-proxy"
Oct 30 21:03:41.493110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.493057    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main"
Oct 30 21:03:41.493110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.493064    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main"
Oct 30 21:03:41.493340 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.493200    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d4df14-1364-4b11-8e9e-5adb5289b607" containerName="main"
Oct 30 21:03:41.493340 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.493218    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02ebacf-ac9c-4d7e-841e-81d36d8631d1" containerName="revoke-keys"
Oct 30 21:03:41.498401 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.498374    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.572471 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.572434    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:41.577022 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.576988    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"]
Oct 30 21:03:41.602373 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.602336    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:03:41.602373 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.602360    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:41.602586 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.602387    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:41.606996 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.606968    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.607145 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.607007    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.607216 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.607182    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlt8g\" (UniqueName: \"kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.708140 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.708103    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.708140 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.708143    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.708386 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.708267    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlt8g\" (UniqueName: \"kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.709349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.709042    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.711315 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.711277    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.766897 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.766823    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlt8g\" (UniqueName: \"kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g\") pod \"maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:41.822488 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:41.822456    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:42.231484 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.231453    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"]
Oct 30 21:03:42.232553 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:03:42.232529    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070822c4_a017_4cf5_bc53_54838808828c.slice/crio-98bacfab9dd3858cf6acce77c2116f622c230a9d63853451a523fffa3b61409d WatchSource:0}: Error finding container 98bacfab9dd3858cf6acce77c2116f622c230a9d63853451a523fffa3b61409d: Status 404 returned error can't find the container with id 98bacfab9dd3858cf6acce77c2116f622c230a9d63853451a523fffa3b61409d
Oct 30 21:03:42.281109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.281076    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"]
Oct 30 21:03:42.286487 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.286462    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.345574 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.345540    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"]
Oct 30 21:03:42.416753 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.416693    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.416924 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.416863    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.416924 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.416899    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtmn\" (UniqueName: \"kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.518245 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.518144    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.518245 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.518220    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtmn\" (UniqueName: \"kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.518733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.518372    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.519060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.519039    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.520668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.520649    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.574873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.574838    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtmn\" (UniqueName: \"kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn\") pod \"maas-api-revoke-sub-79127023-n8s5m\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.604552 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.604522    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:42.986495 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.986458    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6" event={"ID":"070822c4-a017-4cf5-bc53-54838808828c","Type":"ContainerStarted","Data":"b2058628b69e8de5eaf911992fa1e115b12c30b018b751e0aa27658d97906cee"}
Oct 30 21:03:42.986667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:42.986516    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6" event={"ID":"070822c4-a017-4cf5-bc53-54838808828c","Type":"ContainerStarted","Data":"98bacfab9dd3858cf6acce77c2116f622c230a9d63853451a523fffa3b61409d"}
Oct 30 21:03:43.037039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.037013    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"]
Oct 30 21:03:43.039707 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:03:43.039679    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b07256_7360_46c1_9e40_ac72b54e3d38.slice/crio-b5cd37dfd302cf5de7b208d4d1cd183608682764a91bd96eab255eaa779e3d0f WatchSource:0}: Error finding container b5cd37dfd302cf5de7b208d4d1cd183608682764a91bd96eab255eaa779e3d0f: Status 404 returned error can't find the container with id b5cd37dfd302cf5de7b208d4d1cd183608682764a91bd96eab255eaa779e3d0f
Oct 30 21:03:43.100335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.100286    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6" podStartSLOduration=2.1002662069999998 podStartE2EDuration="2.100266207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:43.099347432 +0000 UTC m=+2603.700149392" watchObservedRunningTime="2026-10-30 21:03:43.100266207 +0000 UTC m=+2603.701068170"
Oct 30 21:03:43.995424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.995381    5967 generic.go:356] "Generic (PLEG): container finished" podID="22b07256-7360-46c1-9e40-ac72b54e3d38" containerID="e5fdd7159505c438dd7952b7f4b5f1654fdc64910b363e8d2db8322e9687dd6f" exitCode=0
Oct 30 21:03:43.995852 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.995471    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m" event={"ID":"22b07256-7360-46c1-9e40-ac72b54e3d38","Type":"ContainerDied","Data":"e5fdd7159505c438dd7952b7f4b5f1654fdc64910b363e8d2db8322e9687dd6f"}
Oct 30 21:03:43.995852 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.995523    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m" event={"ID":"22b07256-7360-46c1-9e40-ac72b54e3d38","Type":"ContainerStarted","Data":"b5cd37dfd302cf5de7b208d4d1cd183608682764a91bd96eab255eaa779e3d0f"}
Oct 30 21:03:43.997141 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.997117    5967 generic.go:356] "Generic (PLEG): container finished" podID="070822c4-a017-4cf5-bc53-54838808828c" containerID="b2058628b69e8de5eaf911992fa1e115b12c30b018b751e0aa27658d97906cee" exitCode=0
Oct 30 21:03:43.997264 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:43.997193    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6" event={"ID":"070822c4-a017-4cf5-bc53-54838808828c","Type":"ContainerDied","Data":"b2058628b69e8de5eaf911992fa1e115b12c30b018b751e0aa27658d97906cee"}
Oct 30 21:03:45.185499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.185475    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:45.188864 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.188843    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:45.255096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255017    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca\") pod \"070822c4-a017-4cf5-bc53-54838808828c\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") "
Oct 30 21:03:45.255096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255065    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca\") pod \"070822c4-a017-4cf5-bc53-54838808828c\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") "
Oct 30 21:03:45.255277 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255164    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn\" (UniqueName: \"kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn\") pod \"22b07256-7360-46c1-9e40-ac72b54e3d38\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") "
Oct 30 21:03:45.255324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255310    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca\") pod \"22b07256-7360-46c1-9e40-ac72b54e3d38\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") "
Oct 30 21:03:45.255380 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255366    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca\") pod \"22b07256-7360-46c1-9e40-ac72b54e3d38\" (UID: \"22b07256-7360-46c1-9e40-ac72b54e3d38\") "
Oct 30 21:03:45.255478 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255459    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g\" (UniqueName: \"kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g\") pod \"070822c4-a017-4cf5-bc53-54838808828c\" (UID: \"070822c4-a017-4cf5-bc53-54838808828c\") "
Oct 30 21:03:45.255619 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255451    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca" pod "070822c4-a017-4cf5-bc53-54838808828c" (UID: "070822c4-a017-4cf5-bc53-54838808828c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:45.255726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.255699    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca" pod "22b07256-7360-46c1-9e40-ac72b54e3d38" (UID: "22b07256-7360-46c1-9e40-ac72b54e3d38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:45.256185 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.256161    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070822c4-a017-4cf5-bc53-54838808828c-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.256272 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.256191    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.257518 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.257491    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca" pod "22b07256-7360-46c1-9e40-ac72b54e3d38" (UID: "22b07256-7360-46c1-9e40-ac72b54e3d38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:45.257628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.257523    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca" pod "070822c4-a017-4cf5-bc53-54838808828c" (UID: "070822c4-a017-4cf5-bc53-54838808828c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:45.257688 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.257630    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn" pod "22b07256-7360-46c1-9e40-ac72b54e3d38" (UID: "22b07256-7360-46c1-9e40-ac72b54e3d38"). InnerVolumeSpecName "kube-api-access-fvtmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:45.257688 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.257650    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g" pod "070822c4-a017-4cf5-bc53-54838808828c" (UID: "070822c4-a017-4cf5-bc53-54838808828c"). InnerVolumeSpecName "kube-api-access-hlt8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:45.357505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.357472    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b07256-7360-46c1-9e40-ac72b54e3d38-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.357505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.357502    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlt8g\" (UniqueName: \"kubernetes.io/projected/070822c4-a017-4cf5-bc53-54838808828c-kube-api-access-hlt8g\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.357505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.357513    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070822c4-a017-4cf5-bc53-54838808828c-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.357758 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.357524    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtmn\" (UniqueName: \"kubernetes.io/projected/22b07256-7360-46c1-9e40-ac72b54e3d38-kube-api-access-fvtmn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.531637 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.531552    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:45.531844 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.531552    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:45.976161 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:45.976124    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"]
Oct 30 21:03:46.013524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:46.013492    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"
Oct 30 21:03:46.013728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:46.013526    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m" event={"ID":"22b07256-7360-46c1-9e40-ac72b54e3d38","Type":"ContainerDied","Data":"b5cd37dfd302cf5de7b208d4d1cd183608682764a91bd96eab255eaa779e3d0f"}
Oct 30 21:03:46.013728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:46.013575    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5cd37dfd302cf5de7b208d4d1cd183608682764a91bd96eab255eaa779e3d0f"
Oct 30 21:03:46.015301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:46.015282    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"
Oct 30 21:03:46.015431 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:46.015373    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6" event={"ID":"070822c4-a017-4cf5-bc53-54838808828c","Type":"ContainerDied","Data":"98bacfab9dd3858cf6acce77c2116f622c230a9d63853451a523fffa3b61409d"}
Oct 30 21:03:46.015431 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:46.015419    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bacfab9dd3858cf6acce77c2116f622c230a9d63853451a523fffa3b61409d"
Oct 30 21:03:48.186666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.186631    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"]
Oct 30 21:03:48.187238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.187224    5967 cpu_manager.go:414] "containerMap: removing container" podUID="070822c4-a017-4cf5-bc53-54838808828c" containerName="revoke-keys"
Oct 30 21:03:48.187283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.187239    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070822c4-a017-4cf5-bc53-54838808828c" containerName="revoke-keys"
Oct 30 21:03:48.187283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.187276    5967 cpu_manager.go:414] "containerMap: removing container" podUID="22b07256-7360-46c1-9e40-ac72b54e3d38" containerName="revoke-keys"
Oct 30 21:03:48.187283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.187281    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b07256-7360-46c1-9e40-ac72b54e3d38" containerName="revoke-keys"
Oct 30 21:03:48.187377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.187353    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="070822c4-a017-4cf5-bc53-54838808828c" containerName="revoke-keys"
Oct 30 21:03:48.187377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.187368    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b07256-7360-46c1-9e40-ac72b54e3d38" containerName="revoke-keys"
Oct 30 21:03:48.190933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.190918    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.239008 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.238979    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"]
Oct 30 21:03:48.242909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.242886    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.243498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.243476    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:48.246946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.246919    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"]
Oct 30 21:03:48.285796 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.285764    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:03:48.285796 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.285795    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:48.290618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.290585    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.290810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.290657    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgmk\" (UniqueName: \"kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.290897 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.290823    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.292623 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.292603    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:48.300529 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.300506    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"]
Oct 30 21:03:48.391685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.391651    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.391893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.391772    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.391893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.391793    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmvl\" (UniqueName: \"kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.391893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.391812    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.392081 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.391940    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.392081 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.391981    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgmk\" (UniqueName: \"kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.392533 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.392516    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.394157 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.394136    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.449524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.449460    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgmk\" (UniqueName: \"kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk\") pod \"maas-api-revoke-sub-59c55c59-kzhvv\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.493024 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.492990    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.493193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.493034    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmvl\" (UniqueName: \"kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.493193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.493083    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.493714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.493695    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.495402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.495384    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.504426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.504406    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:48.552909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.552875    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmvl\" (UniqueName: \"kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl\") pod \"maas-api-revoke-sub-66d7658e-cvvpg\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.559054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.558931    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:48.881156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.881127    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"]
Oct 30 21:03:48.882710 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:03:48.882685    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870e547d_4408_4af0_9375_ebd97e1c4ba4.slice/crio-80e3c646248440acd905610afb7bca871407d1cd78a4e6da80001d700057e0ed WatchSource:0}: Error finding container 80e3c646248440acd905610afb7bca871407d1cd78a4e6da80001d700057e0ed: Status 404 returned error can't find the container with id 80e3c646248440acd905610afb7bca871407d1cd78a4e6da80001d700057e0ed
Oct 30 21:03:48.982660 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:48.982624    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"]
Oct 30 21:03:48.987691 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:03:48.987667    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047724ee_8819_434f_8a6d_729958365e61.slice/crio-89b04800f504cab320edc459d29c227a06e4c5318ac445942fc0d6863bce1c56 WatchSource:0}: Error finding container 89b04800f504cab320edc459d29c227a06e4c5318ac445942fc0d6863bce1c56: Status 404 returned error can't find the container with id 89b04800f504cab320edc459d29c227a06e4c5318ac445942fc0d6863bce1c56
Oct 30 21:03:49.035701 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:49.035665    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg" event={"ID":"047724ee-8819-434f-8a6d-729958365e61","Type":"ContainerStarted","Data":"89b04800f504cab320edc459d29c227a06e4c5318ac445942fc0d6863bce1c56"}
Oct 30 21:03:49.037041 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:49.037018    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv" event={"ID":"870e547d-4408-4af0-9375-ebd97e1c4ba4","Type":"ContainerStarted","Data":"80e3c646248440acd905610afb7bca871407d1cd78a4e6da80001d700057e0ed"}
Oct 30 21:03:50.044790 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:50.044679    5967 generic.go:356] "Generic (PLEG): container finished" podID="047724ee-8819-434f-8a6d-729958365e61" containerID="908670df4950c37b93917bc0901abeb7610f731dc9cbdf0fc22fe2f7c3aebbf8" exitCode=0
Oct 30 21:03:50.044790 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:50.044760    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg" event={"ID":"047724ee-8819-434f-8a6d-729958365e61","Type":"ContainerDied","Data":"908670df4950c37b93917bc0901abeb7610f731dc9cbdf0fc22fe2f7c3aebbf8"}
Oct 30 21:03:50.050496 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:50.050468    5967 generic.go:356] "Generic (PLEG): container finished" podID="870e547d-4408-4af0-9375-ebd97e1c4ba4" containerID="b3e15331a8cbe8f7a6888c636e7ffb2e299002da338196f3b7852f33301c7f13" exitCode=0
Oct 30 21:03:50.050635 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:50.050600    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv" event={"ID":"870e547d-4408-4af0-9375-ebd97e1c4ba4","Type":"ContainerDied","Data":"b3e15331a8cbe8f7a6888c636e7ffb2e299002da338196f3b7852f33301c7f13"}
Oct 30 21:03:51.205649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.205626    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:51.232420 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.232396    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:51.326266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326183    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca\") pod \"047724ee-8819-434f-8a6d-729958365e61\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") "
Oct 30 21:03:51.326421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326280    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca\") pod \"047724ee-8819-434f-8a6d-729958365e61\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") "
Oct 30 21:03:51.326421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326340    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca\") pod \"870e547d-4408-4af0-9375-ebd97e1c4ba4\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") "
Oct 30 21:03:51.326421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326390    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl\" (UniqueName: \"kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl\") pod \"047724ee-8819-434f-8a6d-729958365e61\" (UID: \"047724ee-8819-434f-8a6d-729958365e61\") "
Oct 30 21:03:51.326571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326469    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca\") pod \"870e547d-4408-4af0-9375-ebd97e1c4ba4\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") "
Oct 30 21:03:51.326630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326569    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk\" (UniqueName: \"kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk\") pod \"870e547d-4408-4af0-9375-ebd97e1c4ba4\" (UID: \"870e547d-4408-4af0-9375-ebd97e1c4ba4\") "
Oct 30 21:03:51.326681 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326586    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca" pod "047724ee-8819-434f-8a6d-729958365e61" (UID: "047724ee-8819-434f-8a6d-729958365e61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:51.326775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.326753    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca" pod "870e547d-4408-4af0-9375-ebd97e1c4ba4" (UID: "870e547d-4408-4af0-9375-ebd97e1c4ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:51.327292 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.327265    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047724ee-8819-434f-8a6d-729958365e61-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:51.327292 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.327294    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:51.329133 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.328867    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl" pod "047724ee-8819-434f-8a6d-729958365e61" (UID: "047724ee-8819-434f-8a6d-729958365e61"). InnerVolumeSpecName "kube-api-access-njmvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:51.329133 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.328985    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk" pod "870e547d-4408-4af0-9375-ebd97e1c4ba4" (UID: "870e547d-4408-4af0-9375-ebd97e1c4ba4"). InnerVolumeSpecName "kube-api-access-jqgmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:51.329255 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.329173    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca" pod "047724ee-8819-434f-8a6d-729958365e61" (UID: "047724ee-8819-434f-8a6d-729958365e61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:51.329727 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.329698    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca" pod "870e547d-4408-4af0-9375-ebd97e1c4ba4" (UID: "870e547d-4408-4af0-9375-ebd97e1c4ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:51.428675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.428644    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgmk\" (UniqueName: \"kubernetes.io/projected/870e547d-4408-4af0-9375-ebd97e1c4ba4-kube-api-access-jqgmk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:51.428675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.428673    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047724ee-8819-434f-8a6d-729958365e61-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:51.428675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.428682    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njmvl\" (UniqueName: \"kubernetes.io/projected/047724ee-8819-434f-8a6d-729958365e61-kube-api-access-njmvl\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:51.428926 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:51.428693    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870e547d-4408-4af0-9375-ebd97e1c4ba4-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:52.064689 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.064659    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"
Oct 30 21:03:52.064940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.064660    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg" event={"ID":"047724ee-8819-434f-8a6d-729958365e61","Type":"ContainerDied","Data":"89b04800f504cab320edc459d29c227a06e4c5318ac445942fc0d6863bce1c56"}
Oct 30 21:03:52.064940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.064828    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89b04800f504cab320edc459d29c227a06e4c5318ac445942fc0d6863bce1c56"
Oct 30 21:03:52.066080 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.066056    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv" event={"ID":"870e547d-4408-4af0-9375-ebd97e1c4ba4","Type":"ContainerDied","Data":"80e3c646248440acd905610afb7bca871407d1cd78a4e6da80001d700057e0ed"}
Oct 30 21:03:52.066080 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.066074    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"
Oct 30 21:03:52.066244 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.066097    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e3c646248440acd905610afb7bca871407d1cd78a4e6da80001d700057e0ed"
Oct 30 21:03:52.524474 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:52.524442    5967 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954ad950_625d_4ae9_b98f_9b8c54017993.slice/crio-conmon-81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:03:52.711614 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.711552    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 21:03:52.843308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.843278    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt\" (UniqueName: \"kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt\") pod \"954ad950-625d-4ae9-b98f-9b8c54017993\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") "
Oct 30 21:03:52.844306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.843436    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume\") pod \"954ad950-625d-4ae9-b98f-9b8c54017993\" (UID: \"954ad950-625d-4ae9-b98f-9b8c54017993\") "
Oct 30 21:03:52.844306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.843872    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume" pod "954ad950-625d-4ae9-b98f-9b8c54017993" (UID: "954ad950-625d-4ae9-b98f-9b8c54017993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:52.845971 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.845939    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt" pod "954ad950-625d-4ae9-b98f-9b8c54017993" (UID: "954ad950-625d-4ae9-b98f-9b8c54017993"). InnerVolumeSpecName "kube-api-access-4bzxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:52.945610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.945577    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzxt\" (UniqueName: \"kubernetes.io/projected/954ad950-625d-4ae9-b98f-9b8c54017993-kube-api-access-4bzxt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:52.945610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.945608    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954ad950-625d-4ae9-b98f-9b8c54017993-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:52.960150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.960125    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 21:03:52.994025 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:52.993991    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"]
Oct 30 21:03:53.046833 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.046799    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv\" (UniqueName: \"kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv\") pod \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") "
Oct 30 21:03:53.047017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.046934    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume\") pod \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\" (UID: \"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d\") "
Oct 30 21:03:53.047300 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.047275    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume" pod "e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" (UID: "e84156bc-3e2c-46c4-9c5f-ad5d2e91342d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:53.049210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.049192    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv" pod "e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" (UID: "e84156bc-3e2c-46c4-9c5f-ad5d2e91342d"). InnerVolumeSpecName "kube-api-access-6xkfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:53.073427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.073394    5967 generic.go:356] "Generic (PLEG): container finished" podID="954ad950-625d-4ae9-b98f-9b8c54017993" containerID="81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae" exitCode=0
Oct 30 21:03:53.073576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.073481    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"
Oct 30 21:03:53.073576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.073489    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" event={"ID":"954ad950-625d-4ae9-b98f-9b8c54017993","Type":"ContainerDied","Data":"81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae"}
Oct 30 21:03:53.073576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.073557    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx" event={"ID":"954ad950-625d-4ae9-b98f-9b8c54017993","Type":"ContainerDied","Data":"14eb429d161a3a87056b0dbd863032e355f34ab35622f7ff822cb53ebbb6d648"}
Oct 30 21:03:53.073948 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.073589    5967 scope.go:122] "RemoveContainer" containerID="81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae"
Oct 30 21:03:53.082309 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.082281    5967 generic.go:356] "Generic (PLEG): container finished" podID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerID="ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e" exitCode=0
Oct 30 21:03:53.082430 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.082387    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" event={"ID":"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d","Type":"ContainerDied","Data":"ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e"}
Oct 30 21:03:53.082503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.082474    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl" event={"ID":"e84156bc-3e2c-46c4-9c5f-ad5d2e91342d","Type":"ContainerDied","Data":"0ec1e503070c8c50794a073aa122437d1e4796c70f99cc06a6fcda53a5bc9dc0"}
Oct 30 21:03:53.082566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.082528    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"
Oct 30 21:03:53.093705 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.093687    5967 scope.go:122] "RemoveContainer" containerID="81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae"
Oct 30 21:03:53.094076 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:53.094059    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae\": container with ID starting with 81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae not found: ID does not exist" containerID="81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae"
Oct 30 21:03:53.094143 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.094085    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae"} err="failed to get container status \"81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae\": rpc error: code = NotFound desc = could not find container \"81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae\": container with ID starting with 81493538e481a5e2104c47d703d60128d7e42e588dd2afe05ae694126ba5c1ae not found: ID does not exist"
Oct 30 21:03:53.094143 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.094101    5967 scope.go:122] "RemoveContainer" containerID="ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e"
Oct 30 21:03:53.108690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.108664    5967 scope.go:122] "RemoveContainer" containerID="ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e"
Oct 30 21:03:53.109067 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:03:53.109047    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e\": container with ID starting with ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e not found: ID does not exist" containerID="ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e"
Oct 30 21:03:53.109127 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.109076    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e"} err="failed to get container status \"ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e\": rpc error: code = NotFound desc = could not find container \"ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e\": container with ID starting with ccff90dc58096bca9022f80b2e911750edd6e2c724b7c3aceddbf6239312191e not found: ID does not exist"
Oct 30 21:03:53.148397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.148371    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:53.148397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.148392    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xkfv\" (UniqueName: \"kubernetes.io/projected/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d-kube-api-access-6xkfv\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:53.355862 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.355763    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"]
Oct 30 21:03:53.358407 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.358380    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl"]
Oct 30 21:03:53.464759 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.464708    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"]
Oct 30 21:03:53.470864 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.470835    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx"]
Oct 30 21:03:53.543152 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.543122    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" path="/var/lib/kubelet/pods/954ad950-625d-4ae9-b98f-9b8c54017993/volumes"
Oct 30 21:03:53.543577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:53.543564    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" path="/var/lib/kubelet/pods/e84156bc-3e2c-46c4-9c5f-ad5d2e91342d/volumes"
Oct 30 21:03:57.994270 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:57.994227    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw" podUID="fc49c7ea-6e79-464e-9914-d3a42d35ea5d" containerName="payload-processing" containerID="cri-o://ab39175a936747356d3a6f3b91a28ee69f84e296903d9c4169e3576081dcfb82" gracePeriod=25
Oct 30 21:03:58.119070 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.119030    5967 generic.go:356] "Generic (PLEG): container finished" podID="fc49c7ea-6e79-464e-9914-d3a42d35ea5d" containerID="ab39175a936747356d3a6f3b91a28ee69f84e296903d9c4169e3576081dcfb82" exitCode=0
Oct 30 21:03:58.119279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.119093    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw" event={"ID":"fc49c7ea-6e79-464e-9914-d3a42d35ea5d","Type":"ContainerDied","Data":"ab39175a936747356d3a6f3b91a28ee69f84e296903d9c4169e3576081dcfb82"}
Oct 30 21:03:58.271832 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.271806    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 21:03:58.421341 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.421304    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume\") pod \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") "
Oct 30 21:03:58.421637 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.421425    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6\" (UniqueName: \"kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6\") pod \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\" (UID: \"fc49c7ea-6e79-464e-9914-d3a42d35ea5d\") "
Oct 30 21:03:58.421794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.421765    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume" pod "fc49c7ea-6e79-464e-9914-d3a42d35ea5d" (UID: "fc49c7ea-6e79-464e-9914-d3a42d35ea5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:58.423717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.423664    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6" pod "fc49c7ea-6e79-464e-9914-d3a42d35ea5d" (UID: "fc49c7ea-6e79-464e-9914-d3a42d35ea5d"). InnerVolumeSpecName "kube-api-access-r2bj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:58.522981 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.522858    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:58.522981 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:58.522895    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bj6\" (UniqueName: \"kubernetes.io/projected/fc49c7ea-6e79-464e-9914-d3a42d35ea5d-kube-api-access-r2bj6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:59.126622 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:59.126592    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw" event={"ID":"fc49c7ea-6e79-464e-9914-d3a42d35ea5d","Type":"ContainerDied","Data":"9727cea8becf79047f694c022e0d9920ce58f0aa6d56013bc9eef147d3e10b3b"}
Oct 30 21:03:59.126622 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:59.126623    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"
Oct 30 21:03:59.127194 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:59.126647    5967 scope.go:122] "RemoveContainer" containerID="ab39175a936747356d3a6f3b91a28ee69f84e296903d9c4169e3576081dcfb82"
Oct 30 21:03:59.290184 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:59.290147    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"]
Oct 30 21:03:59.339240 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:59.339212    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw"]
Oct 30 21:03:59.542196 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:03:59.542163    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc49c7ea-6e79-464e-9914-d3a42d35ea5d" path="/var/lib/kubelet/pods/fc49c7ea-6e79-464e-9914-d3a42d35ea5d/volumes"
Oct 30 21:04:00.997848 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:00.997806    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main" containerID="cri-o://3524f0c22bc7bf5d870c74c33dbe365b4d57157b6ee060db8fe561d794abf9d1" gracePeriod=45
Oct 30 21:04:01.142229 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.142199    5967 generic.go:356] "Generic (PLEG): container finished" podID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerID="3524f0c22bc7bf5d870c74c33dbe365b4d57157b6ee060db8fe561d794abf9d1" exitCode=0
Oct 30 21:04:01.142426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.142278    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" event={"ID":"6889cfe8-8a85-43c2-b07a-3902ac8d1d97","Type":"ContainerDied","Data":"3524f0c22bc7bf5d870c74c33dbe365b4d57157b6ee060db8fe561d794abf9d1"}
Oct 30 21:04:01.263210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.263150    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 21:04:01.353072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353042    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm\") pod \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") "
Oct 30 21:04:01.353072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353071    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home\") pod \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") "
Oct 30 21:04:01.353325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353138    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj\" (UniqueName: \"kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj\") pod \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") "
Oct 30 21:04:01.353325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353156    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache\") pod \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") "
Oct 30 21:04:01.353325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353266    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir\") pod \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") "
Oct 30 21:04:01.353325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353292    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs\" (UniqueName: \"kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs\") pod \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\" (UID: \"6889cfe8-8a85-43c2-b07a-3902ac8d1d97\") "
Oct 30 21:04:01.353555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353363    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home" pod "6889cfe8-8a85-43c2-b07a-3902ac8d1d97" (UID: "6889cfe8-8a85-43c2-b07a-3902ac8d1d97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.353555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353487    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache" pod "6889cfe8-8a85-43c2-b07a-3902ac8d1d97" (UID: "6889cfe8-8a85-43c2-b07a-3902ac8d1d97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.353638 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353565    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir" pod "6889cfe8-8a85-43c2-b07a-3902ac8d1d97" (UID: "6889cfe8-8a85-43c2-b07a-3902ac8d1d97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.353756 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353726    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.353821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353762    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.353821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.353797    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.355547 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.355525    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs" pod "6889cfe8-8a85-43c2-b07a-3902ac8d1d97" (UID: "6889cfe8-8a85-43c2-b07a-3902ac8d1d97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:01.355667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.355562    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm" pod "6889cfe8-8a85-43c2-b07a-3902ac8d1d97" (UID: "6889cfe8-8a85-43c2-b07a-3902ac8d1d97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.355667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.355603    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj" pod "6889cfe8-8a85-43c2-b07a-3902ac8d1d97" (UID: "6889cfe8-8a85-43c2-b07a-3902ac8d1d97"). InnerVolumeSpecName "kube-api-access-9rwfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:01.455434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.455405    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.455434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.455430    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.455642 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:01.455444    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwfj\" (UniqueName: \"kubernetes.io/projected/6889cfe8-8a85-43c2-b07a-3902ac8d1d97-kube-api-access-9rwfj\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.979792 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:04:01.979713    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:04:02.150920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:02.150890    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"
Oct 30 21:04:02.151374 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:02.150934    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd" event={"ID":"6889cfe8-8a85-43c2-b07a-3902ac8d1d97","Type":"ContainerDied","Data":"e106c3df1e25b7760b9bf9b1c71d5a89af680fc695b0addb65e16bb27eaca0b0"}
Oct 30 21:04:02.151374 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:02.150996    5967 scope.go:122] "RemoveContainer" containerID="3524f0c22bc7bf5d870c74c33dbe365b4d57157b6ee060db8fe561d794abf9d1"
Oct 30 21:04:02.314792 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:02.314690    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"]
Oct 30 21:04:02.363912 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:02.363883    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/simulator-w3-ae2d38-kserve-646f8fd99-t4fxd"]
Oct 30 21:04:03.556181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:03.556144    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" path="/var/lib/kubelet/pods/6889cfe8-8a85-43c2-b07a-3902ac8d1d97/volumes"
Oct 30 21:04:15.594651 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.594599    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"]
Oct 30 21:04:15.595570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595547    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerName="payload-pre-processing"
Oct 30 21:04:15.595570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595577    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerName="payload-pre-processing"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595611    5967 cpu_manager.go:414] "containerMap: removing container" podUID="047724ee-8819-434f-8a6d-729958365e61" containerName="revoke-keys"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595623    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047724ee-8819-434f-8a6d-729958365e61" containerName="revoke-keys"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595653    5967 cpu_manager.go:414] "containerMap: removing container" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595660    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595672    5967 cpu_manager.go:414] "containerMap: removing container" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" containerName="payload-processing"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595680    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" containerName="payload-processing"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595710    5967 cpu_manager.go:414] "containerMap: removing container" podUID="870e547d-4408-4af0-9375-ebd97e1c4ba4" containerName="revoke-keys"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595717    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870e547d-4408-4af0-9375-ebd97e1c4ba4" containerName="revoke-keys"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595731    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fc49c7ea-6e79-464e-9914-d3a42d35ea5d" containerName="payload-processing"
Oct 30 21:04:15.595785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595754    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc49c7ea-6e79-464e-9914-d3a42d35ea5d" containerName="payload-processing"
Oct 30 21:04:15.596299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595893    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="870e547d-4408-4af0-9375-ebd97e1c4ba4" containerName="revoke-keys"
Oct 30 21:04:15.596299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595905    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc49c7ea-6e79-464e-9914-d3a42d35ea5d" containerName="payload-processing"
Oct 30 21:04:15.596299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595922    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84156bc-3e2c-46c4-9c5f-ad5d2e91342d" containerName="payload-pre-processing"
Oct 30 21:04:15.596299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595936    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="047724ee-8819-434f-8a6d-729958365e61" containerName="revoke-keys"
Oct 30 21:04:15.596299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595948    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="6889cfe8-8a85-43c2-b07a-3902ac8d1d97" containerName="main"
Oct 30 21:04:15.596299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.595958    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="954ad950-625d-4ae9-b98f-9b8c54017993" containerName="payload-processing"
Oct 30 21:04:15.599898 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.599872    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.639467 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.639428    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"]
Oct 30 21:04:15.653985 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.653943    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:04:15.694088 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.694056    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:04:15.694344 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.694051    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:04:15.694463 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.694234    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:04:15.732579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.732544    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.732795 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.732681    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wssc\" (UniqueName: \"kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.732795 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.732704    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.834289 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.834251    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.834506 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.834387    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wssc\" (UniqueName: \"kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.834506 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.834416    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.835171 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.835087    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.836808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.836787    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.897016 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.896924    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wssc\" (UniqueName: \"kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc\") pod \"maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:15.915043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:15.915014    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:16.310608 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:16.310580    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"]
Oct 30 21:04:16.312940 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:04:16.312908    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd14125_3b24_4f96_82df_38582b8216a6.slice/crio-10f9025a28dfffebf6c1de58a3a7e5fb5afbf41ab6206ead3f649f67c5fd9fda WatchSource:0}: Error finding container 10f9025a28dfffebf6c1de58a3a7e5fb5afbf41ab6206ead3f649f67c5fd9fda: Status 404 returned error can't find the container with id 10f9025a28dfffebf6c1de58a3a7e5fb5afbf41ab6206ead3f649f67c5fd9fda
Oct 30 21:04:17.255490 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:17.255451    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7" event={"ID":"4fd14125-3b24-4f96-82df-38582b8216a6","Type":"ContainerStarted","Data":"5fa197d82c422ffbf607f6c196b81a09edb1f87a13e5e50f98f60b30111f9f7a"}
Oct 30 21:04:17.255920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:17.255514    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7" event={"ID":"4fd14125-3b24-4f96-82df-38582b8216a6","Type":"ContainerStarted","Data":"10f9025a28dfffebf6c1de58a3a7e5fb5afbf41ab6206ead3f649f67c5fd9fda"}
Oct 30 21:04:17.399559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:17.399270    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7" podStartSLOduration=2.399247348 podStartE2EDuration="2.399247348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:17.379996504 +0000 UTC m=+2637.980798466" watchObservedRunningTime="2026-10-30 21:04:17.399247348 +0000 UTC m=+2638.000049309"
Oct 30 21:04:18.272846 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:18.272802    5967 generic.go:356] "Generic (PLEG): container finished" podID="4fd14125-3b24-4f96-82df-38582b8216a6" containerID="5fa197d82c422ffbf607f6c196b81a09edb1f87a13e5e50f98f60b30111f9f7a" exitCode=0
Oct 30 21:04:18.273284 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:18.272905    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7" event={"ID":"4fd14125-3b24-4f96-82df-38582b8216a6","Type":"ContainerDied","Data":"5fa197d82c422ffbf607f6c196b81a09edb1f87a13e5e50f98f60b30111f9f7a"}
Oct 30 21:04:19.450146 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.450115    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:19.586965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.586867    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca\") pod \"4fd14125-3b24-4f96-82df-38582b8216a6\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") "
Oct 30 21:04:19.586965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.586926    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca\") pod \"4fd14125-3b24-4f96-82df-38582b8216a6\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") "
Oct 30 21:04:19.587213 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.587072    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc\" (UniqueName: \"kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc\") pod \"4fd14125-3b24-4f96-82df-38582b8216a6\" (UID: \"4fd14125-3b24-4f96-82df-38582b8216a6\") "
Oct 30 21:04:19.587653 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.587366    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca" pod "4fd14125-3b24-4f96-82df-38582b8216a6" (UID: "4fd14125-3b24-4f96-82df-38582b8216a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:19.587653 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.587623    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:19.589309 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.589273    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca" pod "4fd14125-3b24-4f96-82df-38582b8216a6" (UID: "4fd14125-3b24-4f96-82df-38582b8216a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:19.589539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.589513    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc" pod "4fd14125-3b24-4f96-82df-38582b8216a6" (UID: "4fd14125-3b24-4f96-82df-38582b8216a6"). InnerVolumeSpecName "kube-api-access-6wssc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:19.689244 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.689208    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wssc\" (UniqueName: \"kubernetes.io/projected/4fd14125-3b24-4f96-82df-38582b8216a6-kube-api-access-6wssc\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:19.689244 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:19.689244    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd14125-3b24-4f96-82df-38582b8216a6-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:20.289707 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:20.289669    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"
Oct 30 21:04:20.289707 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:20.289684    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7" event={"ID":"4fd14125-3b24-4f96-82df-38582b8216a6","Type":"ContainerDied","Data":"10f9025a28dfffebf6c1de58a3a7e5fb5afbf41ab6206ead3f649f67c5fd9fda"}
Oct 30 21:04:20.289961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:20.289765    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f9025a28dfffebf6c1de58a3a7e5fb5afbf41ab6206ead3f649f67c5fd9fda"
Oct 30 21:04:22.145935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.145861    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"]
Oct 30 21:04:22.146428 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.146415    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4fd14125-3b24-4f96-82df-38582b8216a6" containerName="revoke-keys"
Oct 30 21:04:22.146481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.146430    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd14125-3b24-4f96-82df-38582b8216a6" containerName="revoke-keys"
Oct 30 21:04:22.146529 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.146525    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd14125-3b24-4f96-82df-38582b8216a6" containerName="revoke-keys"
Oct 30 21:04:22.148901 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.148875    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.150812 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.150782    5967 scope.go:122] "RemoveContainer" containerID="c705e278bdfa915d6490077c80b67ab8ecdd3ee58dad713c7b9326e0c5a267a1"
Oct 30 21:04:22.162874 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.162850    5967 scope.go:122] "RemoveContainer" containerID="3ee53b4a6e11376c721aca7e26df4665f4f6972db7fb7f0d5b8bd672b0bced28"
Oct 30 21:04:22.172279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.172253    5967 scope.go:122] "RemoveContainer" containerID="df3783656b3b075bbdfc2a006e124a35a879323347d6ca494403ad3c3b33f8c8"
Oct 30 21:04:22.183303 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.183277    5967 scope.go:122] "RemoveContainer" containerID="1f49c970f3ad55d22c5ee74023b207f0bd643c822de588923e5b5f6b8d2eb872"
Oct 30 21:04:22.192775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.192753    5967 scope.go:122] "RemoveContainer" containerID="75e58682ba2056c6d20565a55cbea508350bc95bd39507c3fa3cdb366cf29e55"
Oct 30 21:04:22.204338 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.203828    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"]
Oct 30 21:04:22.212570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.210373    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.218814 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.213284    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"]
Oct 30 21:04:22.220883 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.220857    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4g84\" (UniqueName: \"kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.220979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.220906    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.220979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.220960    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.247424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.247398    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:04:22.256610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.256590    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:04:22.262503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.262477    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"]
Oct 30 21:04:22.275612 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.275480    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:04:22.275612 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.275574    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:04:22.307468 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.307432    5967 generic.go:356] "Generic (PLEG): container finished" podID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerID="bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3" exitCode=0
Oct 30 21:04:22.307645 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.307511    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" event={"ID":"8128db7f-338a-4298-9ae2-92f650cbc9d6","Type":"ContainerDied","Data":"bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"}
Oct 30 21:04:22.308076 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.308051    5967 scope.go:122] "RemoveContainer" containerID="bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"
Oct 30 21:04:22.309794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.309728    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:04:22.309882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.309836    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:04:22.309882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.309869    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:04:22.322366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.322341    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.322496 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.322432    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.322496 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.322458    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.322496 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.322489    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tlk\" (UniqueName: \"kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.322667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.322643    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4g84\" (UniqueName: \"kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.322773 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.322698    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.323608 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.323585    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.325078 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.325059    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.379015 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.378989    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4g84\" (UniqueName: \"kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84\") pod \"maas-api-revoke-sub-e2aab730-zwjph\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.423543 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.423503    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.423734 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.423557    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.423734 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.423606    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tlk\" (UniqueName: \"kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.424335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.424310    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.426802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.426776    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.470123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.470082    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:22.484620 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.484494    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tlk\" (UniqueName: \"kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk\") pod \"maas-api-revoke-sub-0c2fb2c1-frx7l\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.537961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.537825    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:22.855555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.855525    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"]
Oct 30 21:04:22.856415 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:04:22.856387    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cc7ad9_ec4c_4926_a716_223f338516df.slice/crio-e99f1b979b6d0b7deb9728d27a80ea89f99b131d11445fa03dfee543cdb267ed WatchSource:0}: Error finding container e99f1b979b6d0b7deb9728d27a80ea89f99b131d11445fa03dfee543cdb267ed: Status 404 returned error can't find the container with id e99f1b979b6d0b7deb9728d27a80ea89f99b131d11445fa03dfee543cdb267ed
Oct 30 21:04:22.935596 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:22.935568    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"]
Oct 30 21:04:22.937316 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:04:22.937281    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbc4a55_9a57_461f_b73d_e665c51fc56a.slice/crio-5c3eaef80d1074caf43cf034e006d331f1f7442b1b283778aa4f38e099734ee1 WatchSource:0}: Error finding container 5c3eaef80d1074caf43cf034e006d331f1f7442b1b283778aa4f38e099734ee1: Status 404 returned error can't find the container with id 5c3eaef80d1074caf43cf034e006d331f1f7442b1b283778aa4f38e099734ee1
Oct 30 21:04:23.315450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:23.315421    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph" event={"ID":"01cc7ad9-ec4c-4926-a716-223f338516df","Type":"ContainerStarted","Data":"23aaca8080f37263a6a6d8eea478e0df44abd461d03d2be47c1d7a4127187d42"}
Oct 30 21:04:23.315849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:23.315483    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph" event={"ID":"01cc7ad9-ec4c-4926-a716-223f338516df","Type":"ContainerStarted","Data":"e99f1b979b6d0b7deb9728d27a80ea89f99b131d11445fa03dfee543cdb267ed"}
Oct 30 21:04:23.317538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:23.317516    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" event={"ID":"8128db7f-338a-4298-9ae2-92f650cbc9d6","Type":"ContainerStarted","Data":"6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df"}
Oct 30 21:04:23.318872 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:23.318849    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l" event={"ID":"3cbc4a55-9a57-461f-b73d-e665c51fc56a","Type":"ContainerStarted","Data":"5c3eaef80d1074caf43cf034e006d331f1f7442b1b283778aa4f38e099734ee1"}
Oct 30 21:04:23.429556 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:23.429497    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph" podStartSLOduration=1.429478973 podStartE2EDuration="1.429478973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:23.429115748 +0000 UTC m=+2644.029917731" watchObservedRunningTime="2026-10-30 21:04:23.429478973 +0000 UTC m=+2644.030280935"
Oct 30 21:04:24.215230 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.215196    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:04:24.216593 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.216565    5967 patch_prober.go:29] interesting pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:04:24.216785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.216614    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 21:04:24.327063 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.326975    5967 generic.go:356] "Generic (PLEG): container finished" podID="01cc7ad9-ec4c-4926-a716-223f338516df" containerID="23aaca8080f37263a6a6d8eea478e0df44abd461d03d2be47c1d7a4127187d42" exitCode=0
Oct 30 21:04:24.327486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.327053    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph" event={"ID":"01cc7ad9-ec4c-4926-a716-223f338516df","Type":"ContainerDied","Data":"23aaca8080f37263a6a6d8eea478e0df44abd461d03d2be47c1d7a4127187d42"}
Oct 30 21:04:24.328880 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.328860    5967 generic.go:356] "Generic (PLEG): container finished" podID="3cbc4a55-9a57-461f-b73d-e665c51fc56a" containerID="788eebac4aa134a7fdd1a31aba0b4dc05b06c3192e78e0ddddffb1052e43beb5" exitCode=0
Oct 30 21:04:24.329023 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:24.328916    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l" event={"ID":"3cbc4a55-9a57-461f-b73d-e665c51fc56a","Type":"ContainerDied","Data":"788eebac4aa134a7fdd1a31aba0b4dc05b06c3192e78e0ddddffb1052e43beb5"}
Oct 30 21:04:25.216340 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.216299    5967 patch_prober.go:29] interesting pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:04:25.216575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.216363    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 21:04:25.560991 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.560964    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:25.564211 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.564189    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:25.677541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.668286    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk\" (UniqueName: \"kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk\") pod \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") "
Oct 30 21:04:25.677541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.668384    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca\") pod \"01cc7ad9-ec4c-4926-a716-223f338516df\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") "
Oct 30 21:04:25.677541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.668427    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca\") pod \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") "
Oct 30 21:04:25.677541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.668557    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca\") pod \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\" (UID: \"3cbc4a55-9a57-461f-b73d-e665c51fc56a\") "
Oct 30 21:04:25.677541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.668598    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84\" (UniqueName: \"kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84\") pod \"01cc7ad9-ec4c-4926-a716-223f338516df\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") "
Oct 30 21:04:25.677541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.668633    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca\") pod \"01cc7ad9-ec4c-4926-a716-223f338516df\" (UID: \"01cc7ad9-ec4c-4926-a716-223f338516df\") "
Oct 30 21:04:25.690582 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.687259    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk" pod "3cbc4a55-9a57-461f-b73d-e665c51fc56a" (UID: "3cbc4a55-9a57-461f-b73d-e665c51fc56a"). InnerVolumeSpecName "kube-api-access-b2tlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:25.698195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.697882    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca" pod "3cbc4a55-9a57-461f-b73d-e665c51fc56a" (UID: "3cbc4a55-9a57-461f-b73d-e665c51fc56a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:25.712872 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.699980    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca" pod "01cc7ad9-ec4c-4926-a716-223f338516df" (UID: "01cc7ad9-ec4c-4926-a716-223f338516df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:25.713305 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.713272    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca" pod "01cc7ad9-ec4c-4926-a716-223f338516df" (UID: "01cc7ad9-ec4c-4926-a716-223f338516df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:25.718139 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.718105    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84" pod "01cc7ad9-ec4c-4926-a716-223f338516df" (UID: "01cc7ad9-ec4c-4926-a716-223f338516df"). InnerVolumeSpecName "kube-api-access-f4g84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:25.718812 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.718782    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca" pod "3cbc4a55-9a57-461f-b73d-e665c51fc56a" (UID: "3cbc4a55-9a57-461f-b73d-e665c51fc56a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:25.792849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.788664    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:25.792849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.788710    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4g84\" (UniqueName: \"kubernetes.io/projected/01cc7ad9-ec4c-4926-a716-223f338516df-kube-api-access-f4g84\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:25.794153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.793123    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:25.794153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.793176    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2tlk\" (UniqueName: \"kubernetes.io/projected/3cbc4a55-9a57-461f-b73d-e665c51fc56a-kube-api-access-b2tlk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:25.794153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.793193    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cc7ad9-ec4c-4926-a716-223f338516df-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:25.794153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:25.793210    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cbc4a55-9a57-461f-b73d-e665c51fc56a-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:26.216045 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.216010    5967 patch_prober.go:29] interesting pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:04:26.216258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.216075    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 21:04:26.345961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.345927    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l" event={"ID":"3cbc4a55-9a57-461f-b73d-e665c51fc56a","Type":"ContainerDied","Data":"5c3eaef80d1074caf43cf034e006d331f1f7442b1b283778aa4f38e099734ee1"}
Oct 30 21:04:26.345961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.345956    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"
Oct 30 21:04:26.346230 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.345982    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3eaef80d1074caf43cf034e006d331f1f7442b1b283778aa4f38e099734ee1"
Oct 30 21:04:26.347664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.347638    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"
Oct 30 21:04:26.347858 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.347671    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph" event={"ID":"01cc7ad9-ec4c-4926-a716-223f338516df","Type":"ContainerDied","Data":"e99f1b979b6d0b7deb9728d27a80ea89f99b131d11445fa03dfee543cdb267ed"}
Oct 30 21:04:26.347858 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.347713    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99f1b979b6d0b7deb9728d27a80ea89f99b131d11445fa03dfee543cdb267ed"
Oct 30 21:04:26.647610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:26.647567    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:04:27.216047 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:27.216009    5967 patch_prober.go:29] interesting pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:04:27.216219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:27.216065    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 21:04:28.216409 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:28.216374    5967 patch_prober.go:29] interesting pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:04:28.216884 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:28.216433    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 21:04:29.219871 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:29.219842    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:04:29.220946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:29.220927    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:04:29.825919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:29.825884    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"]
Oct 30 21:04:29.828783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:29.828757    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-df031b-g7qdj"]
Oct 30 21:04:31.551001 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:31.550970    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40b216f-90f8-42a0-9e62-d4c0733ccf8c" path="/var/lib/kubelet/pods/d40b216f-90f8-42a0-9e62-d4c0733ccf8c/volumes"
Oct 30 21:04:31.980809 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:04:31.980771    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:04:36.989213 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:36.989179    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"]
Oct 30 21:04:36.989655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:36.989411    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy" containerID="cri-o://6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a" gracePeriod=30
Oct 30 21:04:42.252083 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.252050    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 21:04:42.285958 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.285910    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.286148 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286076    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.286148 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286108    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.286432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286161    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.286432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286213    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.286432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286309    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.286432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286353    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:42.286432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286389    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286442    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286470    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286478    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert\") pod \"8324624e-fbd0-4fc1-92c1-05c60d635176\" (UID: \"8324624e-fbd0-4fc1-92c1-05c60d635176\") "
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286644    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286936    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286952    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286953    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.286961    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.287298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.287100    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:42.288431 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.288403    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:42.288521 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.288495    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:42.288645 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.288626    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:04:42.288902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.288884    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27" pod "8324624e-fbd0-4fc1-92c1-05c60d635176" (UID: "8324624e-fbd0-4fc1-92c1-05c60d635176"). InnerVolumeSpecName "kube-api-access-c7g27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:42.387810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.387724    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.387810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.387766    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7g27\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-kube-api-access-c7g27\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.387810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.387780    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.387810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.387790    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8324624e-fbd0-4fc1-92c1-05c60d635176-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.387810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.387798    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.387810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.387806    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8324624e-fbd0-4fc1-92c1-05c60d635176-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:42.455654 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.455612    5967 generic.go:356] "Generic (PLEG): container finished" podID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerID="6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a" exitCode=0
Oct 30 21:04:42.455867 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.455707    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"
Oct 30 21:04:42.455867 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.455756    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" event={"ID":"8324624e-fbd0-4fc1-92c1-05c60d635176","Type":"ContainerDied","Data":"6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a"}
Oct 30 21:04:42.455867 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.455821    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9" event={"ID":"8324624e-fbd0-4fc1-92c1-05c60d635176","Type":"ContainerDied","Data":"3d5f762b3c3208c5a2e12c690ac9f86fd7c25220d5024f388dd03b5f22bdc13b"}
Oct 30 21:04:42.455867 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.455851    5967 scope.go:122] "RemoveContainer" containerID="6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a"
Oct 30 21:04:42.466804 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.466786    5967 scope.go:122] "RemoveContainer" containerID="f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"
Oct 30 21:04:42.475705 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.475687    5967 scope.go:122] "RemoveContainer" containerID="6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a"
Oct 30 21:04:42.475974 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:04:42.475950    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a\": container with ID starting with 6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a not found: ID does not exist" containerID="6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a"
Oct 30 21:04:42.476037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.475981    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a"} err="failed to get container status \"6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a\": rpc error: code = NotFound desc = could not find container \"6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a\": container with ID starting with 6b01050545ef0014aa80e48fcadbdc077e9e793c61a4033925a9ce9d52067f3a not found: ID does not exist"
Oct 30 21:04:42.476037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.475998    5967 scope.go:122] "RemoveContainer" containerID="f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"
Oct 30 21:04:42.476242 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:04:42.476222    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304\": container with ID starting with f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304 not found: ID does not exist" containerID="f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"
Oct 30 21:04:42.476286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.476248    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304"} err="failed to get container status \"f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304\": rpc error: code = NotFound desc = could not find container \"f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304\": container with ID starting with f6484c531eb1bf66387a09390f0576afc35c478cb5118dcbff7069a90df4c304 not found: ID does not exist"
Oct 30 21:04:42.619704 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.619669    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"]
Oct 30 21:04:42.622969 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:42.622945    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9"]
Oct 30 21:04:43.543349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:43.543317    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" path="/var/lib/kubelet/pods/8324624e-fbd0-4fc1-92c1-05c60d635176/volumes"
Oct 30 21:04:54.580982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.580943    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"]
Oct 30 21:04:54.581492 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581478    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy"
Oct 30 21:04:54.581546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581493    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy"
Oct 30 21:04:54.581546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581513    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3cbc4a55-9a57-461f-b73d-e665c51fc56a" containerName="revoke-keys"
Oct 30 21:04:54.581546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581519    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cbc4a55-9a57-461f-b73d-e665c51fc56a" containerName="revoke-keys"
Oct 30 21:04:54.581546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581535    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy"
Oct 30 21:04:54.581546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581540    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy"
Oct 30 21:04:54.581717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581556    5967 cpu_manager.go:414] "containerMap: removing container" podUID="01cc7ad9-ec4c-4926-a716-223f338516df" containerName="revoke-keys"
Oct 30 21:04:54.581717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581561    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cc7ad9-ec4c-4926-a716-223f338516df" containerName="revoke-keys"
Oct 30 21:04:54.581717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581655    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy"
Oct 30 21:04:54.581717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581667    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cbc4a55-9a57-461f-b73d-e665c51fc56a" containerName="revoke-keys"
Oct 30 21:04:54.581717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581676    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8324624e-fbd0-4fc1-92c1-05c60d635176" containerName="istio-proxy"
Oct 30 21:04:54.581717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.581685    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cc7ad9-ec4c-4926-a716-223f338516df" containerName="revoke-keys"
Oct 30 21:04:54.585097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.585081    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.688435 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.688378    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2fa881\"/\"openshift-service-ca.crt\""
Oct 30 21:04:54.688784 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.688728    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2fa881\"/\"kube-root-ca.crt\""
Oct 30 21:04:54.688935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.688837    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2fa881\"/\"default-dockercfg-skhvm\""
Oct 30 21:04:54.688935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.688867    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2fa881\"/\"test-model-2fa881-kserve-self-signed-certs\""
Oct 30 21:04:54.691535 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.691512    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"]
Oct 30 21:04:54.736504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.736466    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.736780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.736590    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.736780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.736628    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.736894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.736809    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.736894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.736832    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.736961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.736924    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czhw\" (UniqueName: \"kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.838655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.838553    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.838655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.838604    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.838956 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.838671    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czhw\" (UniqueName: \"kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.838956 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.838724    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.838956 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.838787    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.838956 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.838827    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.839174 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.839051    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.839234 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.839222    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.839348 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.839329    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.841306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.841284    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.841398 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.841374    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.896938 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.896910    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czhw\" (UniqueName: \"kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw\") pod \"test-model-2fa881-kserve-5cfdbfd5c5-vfrsk\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") " pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:54.897141 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:54.897126    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:55.270944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:55.270913    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"]
Oct 30 21:04:55.272775 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:04:55.272723    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58d3d030_84f5_4478_80eb_54a7e5c09fb7.slice/crio-cb8e118934f23ed9104f55729ace4cb958e07524741d83c3ec5291f4433b1ba7 WatchSource:0}: Error finding container cb8e118934f23ed9104f55729ace4cb958e07524741d83c3ec5291f4433b1ba7: Status 404 returned error can't find the container with id cb8e118934f23ed9104f55729ace4cb958e07524741d83c3ec5291f4433b1ba7
Oct 30 21:04:55.542603 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:55.542506    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk" event={"ID":"58d3d030-84f5-4478-80eb-54a7e5c09fb7","Type":"ContainerStarted","Data":"438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260"}
Oct 30 21:04:55.542603 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:55.542556    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk" event={"ID":"58d3d030-84f5-4478-80eb-54a7e5c09fb7","Type":"ContainerStarted","Data":"cb8e118934f23ed9104f55729ace4cb958e07524741d83c3ec5291f4433b1ba7"}
Oct 30 21:04:55.657962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:55.657902    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk" podStartSLOduration=1.657886352 podStartE2EDuration="1.657886352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:55.65565932 +0000 UTC m=+2676.256461280" watchObservedRunningTime="2026-10-30 21:04:55.657886352 +0000 UTC m=+2676.258688310"
Oct 30 21:04:55.897581 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:55.897506    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:04:59.663395 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:59.663353    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"]
Oct 30 21:04:59.670548 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:04:59.670518    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ae2d38-5gdm6"]
Oct 30 21:05:00.945688 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:00.945652    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"]
Oct 30 21:05:00.995015 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:00.994978    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcef0e-f7pk7"]
Oct 30 21:05:01.543110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:01.543076    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070822c4-a017-4cf5-bc53-54838808828c" path="/var/lib/kubelet/pods/070822c4-a017-4cf5-bc53-54838808828c/volumes"
Oct 30 21:05:01.543519 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:01.543506    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd14125-3b24-4f96-82df-38582b8216a6" path="/var/lib/kubelet/pods/4fd14125-3b24-4f96-82df-38582b8216a6/volumes"
Oct 30 21:05:01.981594 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:05:01.981558    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:05:04.897819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:04.897774    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:05:04.911050 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:04.911024    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:05:05.643468 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:05.643430    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:05:09.127454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.127421    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"]
Oct 30 21:05:09.132928 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.132900    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.219426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.219392    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bvk\" (UniqueName: \"kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.219627 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.219600    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.219674 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.219643    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.234796 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.234764    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:09.234987 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.234799    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:05:09.234987 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.234831    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:05:09.234987 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.234846    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:05:09.237481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.237462    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"]
Oct 30 21:05:09.320610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.320571    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.320610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.320616    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.320944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.320674    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bvk\" (UniqueName: \"kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.321260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.321241    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.323347 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.323317    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.377302 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.377272    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bvk\" (UniqueName: \"kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk\") pod \"maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.447792 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.447670    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:09.822030 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:09.821910    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"]
Oct 30 21:05:09.827098 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:05:09.827059    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a11fbc_b822_4c28_9b25_714b429aef2d.slice/crio-d75c7f88087c6791a78bca5487f73ad069dc3efec13099220edb04d7c01bd526 WatchSource:0}: Error finding container d75c7f88087c6791a78bca5487f73ad069dc3efec13099220edb04d7c01bd526: Status 404 returned error can't find the container with id d75c7f88087c6791a78bca5487f73ad069dc3efec13099220edb04d7c01bd526
Oct 30 21:05:10.045248 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:10.045213    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"]
Oct 30 21:05:10.050075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:10.050038    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478433c7-n7tg9"]
Oct 30 21:05:10.654431 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:10.654390    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp" event={"ID":"d6a11fbc-b822-4c28-9b25-714b429aef2d","Type":"ContainerStarted","Data":"897bc8446388622e780602919b832b2ef9472f64ad7a1a0130a53f7ab82f37c4"}
Oct 30 21:05:10.654995 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:10.654481    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp" event={"ID":"d6a11fbc-b822-4c28-9b25-714b429aef2d","Type":"ContainerStarted","Data":"d75c7f88087c6791a78bca5487f73ad069dc3efec13099220edb04d7c01bd526"}
Oct 30 21:05:10.772617 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:10.772563    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp" podStartSLOduration=1.772548178 podStartE2EDuration="1.772548178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:10.769824799 +0000 UTC m=+2691.370626760" watchObservedRunningTime="2026-10-30 21:05:10.772548178 +0000 UTC m=+2691.373350137"
Oct 30 21:05:11.543682 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:11.543643    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9564c9-806b-4014-aeab-a5904949e987" path="/var/lib/kubelet/pods/5b9564c9-806b-4014-aeab-a5904949e987/volumes"
Oct 30 21:05:11.662266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:11.662237    5967 generic.go:356] "Generic (PLEG): container finished" podID="d6a11fbc-b822-4c28-9b25-714b429aef2d" containerID="897bc8446388622e780602919b832b2ef9472f64ad7a1a0130a53f7ab82f37c4" exitCode=0
Oct 30 21:05:11.662715 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:11.662323    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp" event={"ID":"d6a11fbc-b822-4c28-9b25-714b429aef2d","Type":"ContainerDied","Data":"897bc8446388622e780602919b832b2ef9472f64ad7a1a0130a53f7ab82f37c4"}
Oct 30 21:05:12.817339 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.817308    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:12.864117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.864088    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca\") pod \"d6a11fbc-b822-4c28-9b25-714b429aef2d\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") "
Oct 30 21:05:12.864328 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.864291    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca\") pod \"d6a11fbc-b822-4c28-9b25-714b429aef2d\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") "
Oct 30 21:05:12.864398 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.864325    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk\" (UniqueName: \"kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk\") pod \"d6a11fbc-b822-4c28-9b25-714b429aef2d\" (UID: \"d6a11fbc-b822-4c28-9b25-714b429aef2d\") "
Oct 30 21:05:12.864512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.864484    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca" pod "d6a11fbc-b822-4c28-9b25-714b429aef2d" (UID: "d6a11fbc-b822-4c28-9b25-714b429aef2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:12.864781 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.864718    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.866538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.866514    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca" pod "d6a11fbc-b822-4c28-9b25-714b429aef2d" (UID: "d6a11fbc-b822-4c28-9b25-714b429aef2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:12.866649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.866618    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk" pod "d6a11fbc-b822-4c28-9b25-714b429aef2d" (UID: "d6a11fbc-b822-4c28-9b25-714b429aef2d"). InnerVolumeSpecName "kube-api-access-n2bvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:12.965807 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.965768    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a11fbc-b822-4c28-9b25-714b429aef2d-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.965973 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:12.965815    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2bvk\" (UniqueName: \"kubernetes.io/projected/d6a11fbc-b822-4c28-9b25-714b429aef2d-kube-api-access-n2bvk\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:13.686047 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:13.686017    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"
Oct 30 21:05:13.686232 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:13.686056    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp" event={"ID":"d6a11fbc-b822-4c28-9b25-714b429aef2d","Type":"ContainerDied","Data":"d75c7f88087c6791a78bca5487f73ad069dc3efec13099220edb04d7c01bd526"}
Oct 30 21:05:13.686232 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:13.686132    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75c7f88087c6791a78bca5487f73ad069dc3efec13099220edb04d7c01bd526"
Oct 30 21:05:22.301211 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:22.301170    5967 scope.go:122] "RemoveContainer" containerID="4c1e6139a2b3a34be298a4233906678b9b4d09f4d1e1fcf68b866ace2b427fbb"
Oct 30 21:05:31.980654 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:05:31.980577    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:05:34.050728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:34.050690    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"]
Oct 30 21:05:34.100474 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:34.100436    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"]
Oct 30 21:05:34.102838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:34.102811    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a812710b-d2xgn"]
Oct 30 21:05:34.104789 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:34.104767    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5576b5b-ktjrq"]
Oct 30 21:05:35.542347 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:35.542301    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74847b1-cc9b-40d8-a46e-265a0c1350dd" path="/var/lib/kubelet/pods/c74847b1-cc9b-40d8-a46e-265a0c1350dd/volumes"
Oct 30 21:05:35.542954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:35.542933    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b" path="/var/lib/kubelet/pods/e9900dc7-a20c-448c-a2ae-f33e1ccd4b0b/volumes"
Oct 30 21:05:35.951307 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:35.951223    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"]
Oct 30 21:05:36.000637 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:36.000596    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-11a0b0-gbxfp"]
Oct 30 21:05:37.545715 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:37.545668    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a11fbc-b822-4c28-9b25-714b429aef2d" path="/var/lib/kubelet/pods/d6a11fbc-b822-4c28-9b25-714b429aef2d/volumes"
Oct 30 21:05:43.855791 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.855755    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"]
Oct 30 21:05:43.856315 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.856285    5967 cpu_manager.go:414] "containerMap: removing container" podUID="d6a11fbc-b822-4c28-9b25-714b429aef2d" containerName="revoke-keys"
Oct 30 21:05:43.856315 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.856296    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a11fbc-b822-4c28-9b25-714b429aef2d" containerName="revoke-keys"
Oct 30 21:05:43.856405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.856395    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a11fbc-b822-4c28-9b25-714b429aef2d" containerName="revoke-keys"
Oct 30 21:05:43.861216 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.861195    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:43.941063 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.941030    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:43.941238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.941145    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:43.941238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.941196    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zp8\" (UniqueName: \"kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:43.962919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.962878    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:43.963101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.962926    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:05:43.963101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.962931    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:05:43.963101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.962979    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:05:43.965543 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:43.965522    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"]
Oct 30 21:05:44.042598 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.042562    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.042838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.042625    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.042838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.042669    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zp8\" (UniqueName: \"kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.043296 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.043269    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.045126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.045105    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.099417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.099381    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zp8\" (UniqueName: \"kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.174310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.174222    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:44.549253 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.549223    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"]
Oct 30 21:05:44.551661 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:05:44.551632    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cffaad_703e_4713_a04d_2875aca681d4.slice/crio-bcf8a8c67ce5ecc39b71cbb1480ad94462a4548ea033dd869fb3e7ba02490552 WatchSource:0}: Error finding container bcf8a8c67ce5ecc39b71cbb1480ad94462a4548ea033dd869fb3e7ba02490552: Status 404 returned error can't find the container with id bcf8a8c67ce5ecc39b71cbb1480ad94462a4548ea033dd869fb3e7ba02490552
Oct 30 21:05:44.902817 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.902784    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4" event={"ID":"90cffaad-703e-4713-a04d-2875aca681d4","Type":"ContainerStarted","Data":"d852df9c861d19660d75fb6b69659ba1e4c690a8e345a33f3603c72c178b993d"}
Oct 30 21:05:44.903221 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:44.902845    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4" event={"ID":"90cffaad-703e-4713-a04d-2875aca681d4","Type":"ContainerStarted","Data":"bcf8a8c67ce5ecc39b71cbb1480ad94462a4548ea033dd869fb3e7ba02490552"}
Oct 30 21:05:45.020177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:45.020125    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4" podStartSLOduration=2.020108245 podStartE2EDuration="2.020108245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:45.016487315 +0000 UTC m=+2725.617289275" watchObservedRunningTime="2026-10-30 21:05:45.020108245 +0000 UTC m=+2725.620910204"
Oct 30 21:05:45.910217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:45.910126    5967 generic.go:356] "Generic (PLEG): container finished" podID="90cffaad-703e-4713-a04d-2875aca681d4" containerID="d852df9c861d19660d75fb6b69659ba1e4c690a8e345a33f3603c72c178b993d" exitCode=0
Oct 30 21:05:45.910628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:45.910223    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4" event={"ID":"90cffaad-703e-4713-a04d-2875aca681d4","Type":"ContainerDied","Data":"d852df9c861d19660d75fb6b69659ba1e4c690a8e345a33f3603c72c178b993d"}
Oct 30 21:05:47.066818 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.066789    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:47.180570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.180471    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca\") pod \"90cffaad-703e-4713-a04d-2875aca681d4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") "
Oct 30 21:05:47.181719 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.180840    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca\") pod \"90cffaad-703e-4713-a04d-2875aca681d4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") "
Oct 30 21:05:47.181719 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.181039    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8\" (UniqueName: \"kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8\") pod \"90cffaad-703e-4713-a04d-2875aca681d4\" (UID: \"90cffaad-703e-4713-a04d-2875aca681d4\") "
Oct 30 21:05:47.181719 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.181553    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca" pod "90cffaad-703e-4713-a04d-2875aca681d4" (UID: "90cffaad-703e-4713-a04d-2875aca681d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:47.184269 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.184225    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca" pod "90cffaad-703e-4713-a04d-2875aca681d4" (UID: "90cffaad-703e-4713-a04d-2875aca681d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:47.184383 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.184341    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8" pod "90cffaad-703e-4713-a04d-2875aca681d4" (UID: "90cffaad-703e-4713-a04d-2875aca681d4"). InnerVolumeSpecName "kube-api-access-q9zp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:47.282706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.282676    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9zp8\" (UniqueName: \"kubernetes.io/projected/90cffaad-703e-4713-a04d-2875aca681d4-kube-api-access-q9zp8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:47.282706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.282701    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:47.282706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.282712    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cffaad-703e-4713-a04d-2875aca681d4-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:47.924578 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.924478    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"
Oct 30 21:05:47.924578 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.924504    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4" event={"ID":"90cffaad-703e-4713-a04d-2875aca681d4","Type":"ContainerDied","Data":"bcf8a8c67ce5ecc39b71cbb1480ad94462a4548ea033dd869fb3e7ba02490552"}
Oct 30 21:05:47.924578 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:47.924562    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf8a8c67ce5ecc39b71cbb1480ad94462a4548ea033dd869fb3e7ba02490552"
Oct 30 21:05:48.361464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:48.361425    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"]
Oct 30 21:05:55.415132 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:55.415082    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"]
Oct 30 21:05:55.567821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:05:55.567785    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"]
Oct 30 21:06:00.415810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:00.415759    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerName="payload-processing" containerID="cri-o://70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181" gracePeriod=25
Oct 30 21:06:00.568659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:00.568623    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" containerName="payload-pre-processing" containerID="cri-o://ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c" gracePeriod=25
Oct 30 21:06:01.980293 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:01.980255    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:06:03.383360 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.383315    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main" containerID="cri-o://316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e" gracePeriod=45
Oct 30 21:06:03.582314 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.582275    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:06:03.582489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.582286    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:06:03.641647 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.641586    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 21:06:03.783631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.783593    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home\") pod \"0fd683f1-7b7d-499a-b774-cce9364fb71a\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") "
Oct 30 21:06:03.783869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.783662    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir\") pod \"0fd683f1-7b7d-499a-b774-cce9364fb71a\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") "
Oct 30 21:06:03.783869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.783706    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5\" (UniqueName: \"kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5\") pod \"0fd683f1-7b7d-499a-b774-cce9364fb71a\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") "
Oct 30 21:06:03.783869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.783769    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache\") pod \"0fd683f1-7b7d-499a-b774-cce9364fb71a\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") "
Oct 30 21:06:03.783869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.783818    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm\") pod \"0fd683f1-7b7d-499a-b774-cce9364fb71a\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") "
Oct 30 21:06:03.783869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.783861    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs\") pod \"0fd683f1-7b7d-499a-b774-cce9364fb71a\" (UID: \"0fd683f1-7b7d-499a-b774-cce9364fb71a\") "
Oct 30 21:06:03.784191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.784111    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir" pod "0fd683f1-7b7d-499a-b774-cce9364fb71a" (UID: "0fd683f1-7b7d-499a-b774-cce9364fb71a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.784191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.784141    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache" pod "0fd683f1-7b7d-499a-b774-cce9364fb71a" (UID: "0fd683f1-7b7d-499a-b774-cce9364fb71a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.784516 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.784489    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.784627 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.784522    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.784783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.784716    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home" pod "0fd683f1-7b7d-499a-b774-cce9364fb71a" (UID: "0fd683f1-7b7d-499a-b774-cce9364fb71a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.787009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.786981    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5" pod "0fd683f1-7b7d-499a-b774-cce9364fb71a" (UID: "0fd683f1-7b7d-499a-b774-cce9364fb71a"). InnerVolumeSpecName "kube-api-access-q2gv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:03.787120 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.787081    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs" pod "0fd683f1-7b7d-499a-b774-cce9364fb71a" (UID: "0fd683f1-7b7d-499a-b774-cce9364fb71a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:03.787931 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.787902    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm" pod "0fd683f1-7b7d-499a-b774-cce9364fb71a" (UID: "0fd683f1-7b7d-499a-b774-cce9364fb71a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.885320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.885275    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.885320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.885309    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gv5\" (UniqueName: \"kubernetes.io/projected/0fd683f1-7b7d-499a-b774-cce9364fb71a-kube-api-access-q2gv5\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.885320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.885319    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd683f1-7b7d-499a-b774-cce9364fb71a-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.885320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:03.885328    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd683f1-7b7d-499a-b774-cce9364fb71a-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:04.035144 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.035112    5967 generic.go:356] "Generic (PLEG): container finished" podID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerID="316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e" exitCode=0
Oct 30 21:06:04.035342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.035172    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"
Oct 30 21:06:04.035342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.035195    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" event={"ID":"0fd683f1-7b7d-499a-b774-cce9364fb71a","Type":"ContainerDied","Data":"316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e"}
Oct 30 21:06:04.035342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.035237    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs" event={"ID":"0fd683f1-7b7d-499a-b774-cce9364fb71a","Type":"ContainerDied","Data":"be7a9a1eb523231f1bce055c1608a1a5678ad7d86b6d1ac28b82a4ce13b654b7"}
Oct 30 21:06:04.035342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.035260    5967 scope.go:122] "RemoveContainer" containerID="316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e"
Oct 30 21:06:04.047219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.047194    5967 scope.go:122] "RemoveContainer" containerID="316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e"
Oct 30 21:06:04.047582 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:04.047557    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e\": container with ID starting with 316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e not found: ID does not exist" containerID="316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e"
Oct 30 21:06:04.047676 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.047594    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e"} err="failed to get container status \"316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e\": rpc error: code = NotFound desc = could not find container \"316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e\": container with ID starting with 316cc4b07202c2a289527d617cfa37ded7184b7fe234ba8fc5c162813f446e7e not found: ID does not exist"
Oct 30 21:06:04.198542 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.198508    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"]
Oct 30 21:06:04.247691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:04.247652    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/premium-w0-0a78ac-kserve-7f79767f4b-ljvvs"]
Oct 30 21:06:05.542869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:05.542837    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" path="/var/lib/kubelet/pods/0fd683f1-7b7d-499a-b774-cce9364fb71a/volumes"
Oct 30 21:06:07.543402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.543371    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"]
Oct 30 21:06:07.543934 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.543579    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy" containerID="cri-o://1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e" gracePeriod=30
Oct 30 21:06:07.591232 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.591189    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"]
Oct 30 21:06:07.591455 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.591403    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" containerID="cri-o://5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e" gracePeriod=30
Oct 30 21:06:07.593693 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.593665    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"]
Oct 30 21:06:07.593900 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.593879    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" containerID="cri-o://86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672" gracePeriod=30
Oct 30 21:06:07.595952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.595923    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"]
Oct 30 21:06:07.596130 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:07.596109    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy" containerID="cri-o://7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3" gracePeriod=30
Oct 30 21:06:08.041072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:08.041034    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pdvns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:06:08.041276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:08.041094    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:06:08.109817 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:08.109779    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:06:08.109990 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:08.109844    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:06:09.130619 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.130582    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"]
Oct 30 21:06:09.131150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.131136    5967 cpu_manager.go:414] "containerMap: removing container" podUID="90cffaad-703e-4713-a04d-2875aca681d4" containerName="revoke-keys"
Oct 30 21:06:09.131196 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.131151    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cffaad-703e-4713-a04d-2875aca681d4" containerName="revoke-keys"
Oct 30 21:06:09.131196 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.131164    5967 cpu_manager.go:414] "containerMap: removing container" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main"
Oct 30 21:06:09.131196 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.131170    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main"
Oct 30 21:06:09.131303 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.131254    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cffaad-703e-4713-a04d-2875aca681d4" containerName="revoke-keys"
Oct 30 21:06:09.131303 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.131265    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd683f1-7b7d-499a-b774-cce9364fb71a" containerName="main"
Oct 30 21:06:09.134896 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.134875    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.186965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.186928    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"]
Oct 30 21:06:09.191301 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.191270    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.233868 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.233835    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:06:09.234065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.233903    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:06:09.236315 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.236285    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:06:09.236461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.236322    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:06:09.236628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.236605    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"]
Oct 30 21:06:09.247219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.247155    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"]
Oct 30 21:06:09.252426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.252394    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.252597 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.252498    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtgl\" (UniqueName: \"kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.252702 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.252682    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.353831 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.353798    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.353831 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.353836    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtgl\" (UniqueName: \"kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.354057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.353975    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.354057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.354041    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.354227 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.354157    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.354227 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.354200    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spws9\" (UniqueName: \"kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.354862 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.354844    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.356508 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.356490    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.410176 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.410095    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtgl\" (UniqueName: \"kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl\") pod \"maas-api-revoke-sub-719cb10b-z94gq\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.449190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.449154    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:09.456028 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.456001    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.456154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.456138    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spws9\" (UniqueName: \"kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.456241 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.456226    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.456694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.456671    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.458782 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.458731    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.513778 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.513548    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spws9\" (UniqueName: \"kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9\") pod \"maas-api-revoke-sub-a2b0609d-c8nrs\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.806671 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.806636    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:09.825666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:09.825638    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"]
Oct 30 21:06:09.826707 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:06:09.826682    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbda826e_1568_4840_a766_6d6db04d3ca8.slice/crio-b2cff9e727550486b22712bf462193e95206cfeae04b24f6b0130706b13f9a27 WatchSource:0}: Error finding container b2cff9e727550486b22712bf462193e95206cfeae04b24f6b0130706b13f9a27: Status 404 returned error can't find the container with id b2cff9e727550486b22712bf462193e95206cfeae04b24f6b0130706b13f9a27
Oct 30 21:06:10.079439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:10.079404    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq" event={"ID":"dbda826e-1568-4840-a766-6d6db04d3ca8","Type":"ContainerStarted","Data":"b2cff9e727550486b22712bf462193e95206cfeae04b24f6b0130706b13f9a27"}
Oct 30 21:06:10.186228 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:10.186204    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"]
Oct 30 21:06:10.187933 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:06:10.187905    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3005d396_c394_4ff8_8ecc_dd6d2274adda.slice/crio-4c4eae956c76099ef6e16bfdbcea9cfcb2b978ae7df85664f7e340b6f9efbbf3 WatchSource:0}: Error finding container 4c4eae956c76099ef6e16bfdbcea9cfcb2b978ae7df85664f7e340b6f9efbbf3: Status 404 returned error can't find the container with id 4c4eae956c76099ef6e16bfdbcea9cfcb2b978ae7df85664f7e340b6f9efbbf3
Oct 30 21:06:11.090133 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:11.090080    5967 generic.go:356] "Generic (PLEG): container finished" podID="dbda826e-1568-4840-a766-6d6db04d3ca8" containerID="1ad59ee2addb518abae8a3f39aca1d139e09f08451f03bd05e9d159443bdf59e" exitCode=0
Oct 30 21:06:11.090318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:11.090248    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq" event={"ID":"dbda826e-1568-4840-a766-6d6db04d3ca8","Type":"ContainerDied","Data":"1ad59ee2addb518abae8a3f39aca1d139e09f08451f03bd05e9d159443bdf59e"}
Oct 30 21:06:11.097594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:11.097568    5967 generic.go:356] "Generic (PLEG): container finished" podID="3005d396-c394-4ff8-8ecc-dd6d2274adda" containerID="355c7f1fb5f1fcedb7c89f6a4da19e08b577fa882e0f94b653c8291c270a59d9" exitCode=0
Oct 30 21:06:11.097724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:11.097620    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs" event={"ID":"3005d396-c394-4ff8-8ecc-dd6d2274adda","Type":"ContainerDied","Data":"355c7f1fb5f1fcedb7c89f6a4da19e08b577fa882e0f94b653c8291c270a59d9"}
Oct 30 21:06:11.097724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:11.097659    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs" event={"ID":"3005d396-c394-4ff8-8ecc-dd6d2274adda","Type":"ContainerStarted","Data":"4c4eae956c76099ef6e16bfdbcea9cfcb2b978ae7df85664f7e340b6f9efbbf3"}
Oct 30 21:06:12.274297 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.274269    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:12.276284 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.276265    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:12.394723 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.394644    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca\") pod \"dbda826e-1568-4840-a766-6d6db04d3ca8\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") "
Oct 30 21:06:12.394936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.394724    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca\") pod \"3005d396-c394-4ff8-8ecc-dd6d2274adda\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") "
Oct 30 21:06:12.394936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.394768    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca\") pod \"dbda826e-1568-4840-a766-6d6db04d3ca8\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") "
Oct 30 21:06:12.395076 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.394972    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca\") pod \"3005d396-c394-4ff8-8ecc-dd6d2274adda\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") "
Oct 30 21:06:12.395076 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.395025    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9\" (UniqueName: \"kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9\") pod \"3005d396-c394-4ff8-8ecc-dd6d2274adda\" (UID: \"3005d396-c394-4ff8-8ecc-dd6d2274adda\") "
Oct 30 21:06:12.395186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.395075    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl\" (UniqueName: \"kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl\") pod \"dbda826e-1568-4840-a766-6d6db04d3ca8\" (UID: \"dbda826e-1568-4840-a766-6d6db04d3ca8\") "
Oct 30 21:06:12.395186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.395082    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca" pod "dbda826e-1568-4840-a766-6d6db04d3ca8" (UID: "dbda826e-1568-4840-a766-6d6db04d3ca8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:12.395186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.395146    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca" pod "3005d396-c394-4ff8-8ecc-dd6d2274adda" (UID: "3005d396-c394-4ff8-8ecc-dd6d2274adda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:12.398126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.395563    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.398126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.395590    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.398126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.397387    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9" pod "3005d396-c394-4ff8-8ecc-dd6d2274adda" (UID: "3005d396-c394-4ff8-8ecc-dd6d2274adda"). InnerVolumeSpecName "kube-api-access-spws9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:12.398126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.397521    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca" pod "3005d396-c394-4ff8-8ecc-dd6d2274adda" (UID: "3005d396-c394-4ff8-8ecc-dd6d2274adda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:12.398126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.397830    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca" pod "dbda826e-1568-4840-a766-6d6db04d3ca8" (UID: "dbda826e-1568-4840-a766-6d6db04d3ca8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:12.398880 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.398856    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl" pod "dbda826e-1568-4840-a766-6d6db04d3ca8" (UID: "dbda826e-1568-4840-a766-6d6db04d3ca8"). InnerVolumeSpecName "kube-api-access-tjtgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:12.496324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.496296    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3005d396-c394-4ff8-8ecc-dd6d2274adda-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.496324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.496322    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spws9\" (UniqueName: \"kubernetes.io/projected/3005d396-c394-4ff8-8ecc-dd6d2274adda-kube-api-access-spws9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.496324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.496332    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtgl\" (UniqueName: \"kubernetes.io/projected/dbda826e-1568-4840-a766-6d6db04d3ca8-kube-api-access-tjtgl\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.496565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.496340    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbda826e-1568-4840-a766-6d6db04d3ca8-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.842779 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.839700    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 21:06:12.893102 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.893075    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 21:06:12.931137 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.931071    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 21:06:12.934404 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:12.934384    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 21:06:13.002866 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.002834    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.003075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.002890    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.002909    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.002949    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.003075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.002972    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.003075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003015    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.003075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003047    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003088    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003116    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003143    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003188    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003218    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003234    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003278    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003317    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb\") pod \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\" (UID: \"29698393-dfcb-4a5c-8d99-eba4d45f1aba\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003365    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.003438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003424    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.004004 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003547    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.004004 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003580    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket\") pod \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\" (UID: \"9780fc58-0b21-4bd0-81c4-ca1676765f8c\") "
Oct 30 21:06:13.004004 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.003883    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.004173 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004067    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004286    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004311    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004325    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004349    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004525    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004621    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004642    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.004992    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.005802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.005531    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:13.006562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.005857    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.006562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.005971    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.006562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.006070    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:13.006562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.006086    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.007181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.007160    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.007331 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.007302    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "kube-api-access-z5nsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.008028 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.008009    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo" pod "29698393-dfcb-4a5c-8d99-eba4d45f1aba" (UID: "29698393-dfcb-4a5c-8d99-eba4d45f1aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:13.008434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.008416    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "kube-api-access-w5r52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.009095 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.009080    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token" pod "9780fc58-0b21-4bd0-81c4-ca1676765f8c" (UID: "9780fc58-0b21-4bd0-81c4-ca1676765f8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.105031 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.104995    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105225 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105077    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105225 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105114    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105225 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105172    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105225 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105209    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105255    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105289    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105324    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105352    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105382    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105421    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105462    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105490    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105578    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert\") pod \"e174b0ce-6120-43db-a840-bf69db117b7f\" (UID: \"e174b0ce-6120-43db-a840-bf69db117b7f\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105606    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105632    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105705    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105786    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket\") pod \"34803b24-bb46-4c76-86a9-2a7b646d9160\" (UID: \"34803b24-bb46-4c76-86a9-2a7b646d9160\") "
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105823    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.105945    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.105989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106063    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.106716 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106130    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:13.106716 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106282    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.106716 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106480    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.106716 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106676    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.106716 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106694    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.107024 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.106873    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.107136 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.107111    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:13.108158 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.108131    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:13.108260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.108233    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.108813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.108785    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.109281 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109263    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109281 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109282    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109292    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109300    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109308    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109317    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109330    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e174b0ce-6120-43db-a840-bf69db117b7f-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109344    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109354    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109363    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109371    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109379    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109388    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109403    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109419    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34803b24-bb46-4c76-86a9-2a7b646d9160-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.109427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109423    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "kube-api-access-2bb6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109434    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109479    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109490    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109494    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109519    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29698393-dfcb-4a5c-8d99-eba4d45f1aba-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109533    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109546    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109559    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109575    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29698393-dfcb-4a5c-8d99-eba4d45f1aba-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109590    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nsb\" (UniqueName: \"kubernetes.io/projected/29698393-dfcb-4a5c-8d99-eba4d45f1aba-kube-api-access-z5nsb\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109605    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780fc58-0b21-4bd0-81c4-ca1676765f8c-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109641    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e174b0ce-6120-43db-a840-bf69db117b7f-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109721    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.109831    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5r52\" (UniqueName: \"kubernetes.io/projected/9780fc58-0b21-4bd0-81c4-ca1676765f8c-kube-api-access-w5r52\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.110190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.110162    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo" pod "34803b24-bb46-4c76-86a9-2a7b646d9160" (UID: "34803b24-bb46-4c76-86a9-2a7b646d9160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:13.110986 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.110587    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.111110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.111090    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x" pod "e174b0ce-6120-43db-a840-bf69db117b7f" (UID: "e174b0ce-6120-43db-a840-bf69db117b7f"). InnerVolumeSpecName "kube-api-access-6ll4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.115016 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.114995    5967 generic.go:356] "Generic (PLEG): container finished" podID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerID="86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672" exitCode=0
Oct 30 21:06:13.115121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.115050    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"
Oct 30 21:06:13.115121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.115051    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" event={"ID":"29698393-dfcb-4a5c-8d99-eba4d45f1aba","Type":"ContainerDied","Data":"86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672"}
Oct 30 21:06:13.115121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.115102    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns" event={"ID":"29698393-dfcb-4a5c-8d99-eba4d45f1aba","Type":"ContainerDied","Data":"08df1aa9a7251492dd5b8da31697ebe2e29a4333b239397e1e73bde3f4ea93b0"}
Oct 30 21:06:13.115277 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.115130    5967 scope.go:122] "RemoveContainer" containerID="86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672"
Oct 30 21:06:13.121782 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.121707    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"
Oct 30 21:06:13.121782 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.121725    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq" event={"ID":"dbda826e-1568-4840-a766-6d6db04d3ca8","Type":"ContainerDied","Data":"b2cff9e727550486b22712bf462193e95206cfeae04b24f6b0130706b13f9a27"}
Oct 30 21:06:13.121942 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.121793    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2cff9e727550486b22712bf462193e95206cfeae04b24f6b0130706b13f9a27"
Oct 30 21:06:13.128461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.128436    5967 scope.go:122] "RemoveContainer" containerID="86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672"
Oct 30 21:06:13.129203 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:13.129180    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672\": container with ID starting with 86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672 not found: ID does not exist" containerID="86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672"
Oct 30 21:06:13.129310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.129211    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672"} err="failed to get container status \"86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672\": rpc error: code = NotFound desc = could not find container \"86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672\": container with ID starting with 86a005ef4aa189a4bf83fc2faddfbe85e4b90b286ce2e3214673e91af6f73672 not found: ID does not exist"
Oct 30 21:06:13.129975 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.129878    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"
Oct 30 21:06:13.129975 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.129890    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs" event={"ID":"3005d396-c394-4ff8-8ecc-dd6d2274adda","Type":"ContainerDied","Data":"4c4eae956c76099ef6e16bfdbcea9cfcb2b978ae7df85664f7e340b6f9efbbf3"}
Oct 30 21:06:13.129975 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.129953    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4eae956c76099ef6e16bfdbcea9cfcb2b978ae7df85664f7e340b6f9efbbf3"
Oct 30 21:06:13.132578 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.132557    5967 generic.go:356] "Generic (PLEG): container finished" podID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerID="1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e" exitCode=0
Oct 30 21:06:13.132694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.132600    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"
Oct 30 21:06:13.132694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.132633    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" event={"ID":"9780fc58-0b21-4bd0-81c4-ca1676765f8c","Type":"ContainerDied","Data":"1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e"}
Oct 30 21:06:13.132694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.132680    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v" event={"ID":"9780fc58-0b21-4bd0-81c4-ca1676765f8c","Type":"ContainerDied","Data":"49fe384254d65ec3511d9af80d68447bd933fc36770c291d44b34debe2b2d8a5"}
Oct 30 21:06:13.132694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.132694    5967 scope.go:122] "RemoveContainer" containerID="1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e"
Oct 30 21:06:13.134361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.134339    5967 generic.go:356] "Generic (PLEG): container finished" podID="e174b0ce-6120-43db-a840-bf69db117b7f" containerID="7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3" exitCode=0
Oct 30 21:06:13.134473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.134398    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"
Oct 30 21:06:13.134473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.134420    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" event={"ID":"e174b0ce-6120-43db-a840-bf69db117b7f","Type":"ContainerDied","Data":"7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3"}
Oct 30 21:06:13.134570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.134478    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l" event={"ID":"e174b0ce-6120-43db-a840-bf69db117b7f","Type":"ContainerDied","Data":"d856f4c1652701713c2d8c08e57ab6d3489b1c40f4e32dd669cbae81bf76e368"}
Oct 30 21:06:13.138336 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.138316    5967 generic.go:356] "Generic (PLEG): container finished" podID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerID="5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e" exitCode=0
Oct 30 21:06:13.138427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.138360    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"
Oct 30 21:06:13.138481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.138430    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" event={"ID":"34803b24-bb46-4c76-86a9-2a7b646d9160","Type":"ContainerDied","Data":"5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e"}
Oct 30 21:06:13.138481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.138457    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5" event={"ID":"34803b24-bb46-4c76-86a9-2a7b646d9160","Type":"ContainerDied","Data":"ee46934a231050c641ebbc767f1e503f584d208f8af052034cb535efb797faab"}
Oct 30 21:06:13.144607 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.144585    5967 scope.go:122] "RemoveContainer" containerID="1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e"
Oct 30 21:06:13.144945 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:13.144924    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e\": container with ID starting with 1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e not found: ID does not exist" containerID="1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e"
Oct 30 21:06:13.145026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.144953    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e"} err="failed to get container status \"1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e\": rpc error: code = NotFound desc = could not find container \"1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e\": container with ID starting with 1eeab421e6715790fc27c98579921b64557e467c779c1519374137bf1880408e not found: ID does not exist"
Oct 30 21:06:13.145026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.144971    5967 scope.go:122] "RemoveContainer" containerID="7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3"
Oct 30 21:06:13.155692 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.155676    5967 scope.go:122] "RemoveContainer" containerID="7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3"
Oct 30 21:06:13.155981 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:13.155961    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3\": container with ID starting with 7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3 not found: ID does not exist" containerID="7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3"
Oct 30 21:06:13.156063 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.155993    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3"} err="failed to get container status \"7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3\": rpc error: code = NotFound desc = could not find container \"7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3\": container with ID starting with 7365f247f9cfc06348232c115bebd0c66df0a0b8654b83f842f9b7ec2ce0def3 not found: ID does not exist"
Oct 30 21:06:13.156063 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.156016    5967 scope.go:122] "RemoveContainer" containerID="5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e"
Oct 30 21:06:13.164919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.164900    5967 scope.go:122] "RemoveContainer" containerID="5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e"
Oct 30 21:06:13.165182 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:13.165164    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e\": container with ID starting with 5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e not found: ID does not exist" containerID="5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e"
Oct 30 21:06:13.165231 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.165189    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e"} err="failed to get container status \"5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e\": rpc error: code = NotFound desc = could not find container \"5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e\": container with ID starting with 5a44e71a15f30740dac2c316f0eeac7ff08946aac29583e3eee4cb027c1c290e not found: ID does not exist"
Oct 30 21:06:13.210464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.210391    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.210464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.210416    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34803b24-bb46-4c76-86a9-2a7b646d9160-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.210464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.210426    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ll4x\" (UniqueName: \"kubernetes.io/projected/e174b0ce-6120-43db-a840-bf69db117b7f-kube-api-access-6ll4x\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.210464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.210436    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e174b0ce-6120-43db-a840-bf69db117b7f-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.210464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.210445    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bb6b\" (UniqueName: \"kubernetes.io/projected/34803b24-bb46-4c76-86a9-2a7b646d9160-kube-api-access-2bb6b\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.428905 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.428870    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"]
Oct 30 21:06:13.461715 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.461628    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v"]
Oct 30 21:06:13.543287 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.543247    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" path="/var/lib/kubelet/pods/9780fc58-0b21-4bd0-81c4-ca1676765f8c/volumes"
Oct 30 21:06:13.543633 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.543614    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"]
Oct 30 21:06:13.545457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.545436    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5"]
Oct 30 21:06:13.581571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.581540    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:06:13.581783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.581538    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:06:13.652751 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.652708    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"]
Oct 30 21:06:13.666104 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.666071    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l"]
Oct 30 21:06:13.765052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.765013    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"]
Oct 30 21:06:13.769034 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:13.769007    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns"]
Oct 30 21:06:15.541962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:15.541930    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" path="/var/lib/kubelet/pods/29698393-dfcb-4a5c-8d99-eba4d45f1aba/volumes"
Oct 30 21:06:15.542382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:15.542369    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" path="/var/lib/kubelet/pods/34803b24-bb46-4c76-86a9-2a7b646d9160/volumes"
Oct 30 21:06:15.542799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:15.542784    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" path="/var/lib/kubelet/pods/e174b0ce-6120-43db-a840-bf69db117b7f/volumes"
Oct 30 21:06:20.698700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.698667    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 21:06:20.794021 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.793989    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8\" (UniqueName: \"kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8\") pod \"fbb207bc-4523-44b8-88af-5c038f22a2ef\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") "
Oct 30 21:06:20.794217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.794138    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume\") pod \"fbb207bc-4523-44b8-88af-5c038f22a2ef\" (UID: \"fbb207bc-4523-44b8-88af-5c038f22a2ef\") "
Oct 30 21:06:20.794499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.794478    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume" pod "fbb207bc-4523-44b8-88af-5c038f22a2ef" (UID: "fbb207bc-4523-44b8-88af-5c038f22a2ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:20.796840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.796815    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8" pod "fbb207bc-4523-44b8-88af-5c038f22a2ef" (UID: "fbb207bc-4523-44b8-88af-5c038f22a2ef"). InnerVolumeSpecName "kube-api-access-vjld8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:20.818323 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.818295    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 21:06:20.895367 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.895330    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume\") pod \"9918a6dc-843b-483b-ab96-c50c676c13de\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") "
Oct 30 21:06:20.895584 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.895441    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr\" (UniqueName: \"kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr\") pod \"9918a6dc-843b-483b-ab96-c50c676c13de\" (UID: \"9918a6dc-843b-483b-ab96-c50c676c13de\") "
Oct 30 21:06:20.895675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.895661    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjld8\" (UniqueName: \"kubernetes.io/projected/fbb207bc-4523-44b8-88af-5c038f22a2ef-kube-api-access-vjld8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:20.895714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.895678    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb207bc-4523-44b8-88af-5c038f22a2ef-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:20.895770 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.895714    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume" pod "9918a6dc-843b-483b-ab96-c50c676c13de" (UID: "9918a6dc-843b-483b-ab96-c50c676c13de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:20.897653 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.897631    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr" pod "9918a6dc-843b-483b-ab96-c50c676c13de" (UID: "9918a6dc-843b-483b-ab96-c50c676c13de"). InnerVolumeSpecName "kube-api-access-42tsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:20.996260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.996218    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918a6dc-843b-483b-ab96-c50c676c13de-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:20.996260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:20.996253    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tsr\" (UniqueName: \"kubernetes.io/projected/9918a6dc-843b-483b-ab96-c50c676c13de-kube-api-access-42tsr\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:21.196137 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.196102    5967 generic.go:356] "Generic (PLEG): container finished" podID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerID="70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181" exitCode=0
Oct 30 21:06:21.196356 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.196148    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"
Oct 30 21:06:21.196356 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.196182    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" event={"ID":"fbb207bc-4523-44b8-88af-5c038f22a2ef","Type":"ContainerDied","Data":"70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181"}
Oct 30 21:06:21.196356 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.196233    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k" event={"ID":"fbb207bc-4523-44b8-88af-5c038f22a2ef","Type":"ContainerDied","Data":"e2c480c9c569c655c2d80ec546a3a68b6f82d7c8591634978fb614381e407d85"}
Oct 30 21:06:21.196356 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.196253    5967 scope.go:122] "RemoveContainer" containerID="70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181"
Oct 30 21:06:21.197951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.197930    5967 generic.go:356] "Generic (PLEG): container finished" podID="9918a6dc-843b-483b-ab96-c50c676c13de" containerID="ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c" exitCode=0
Oct 30 21:06:21.198065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.197985    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"
Oct 30 21:06:21.198065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.198014    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" event={"ID":"9918a6dc-843b-483b-ab96-c50c676c13de","Type":"ContainerDied","Data":"ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c"}
Oct 30 21:06:21.198065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.198055    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt" event={"ID":"9918a6dc-843b-483b-ab96-c50c676c13de","Type":"ContainerDied","Data":"360298ed23a9f36b33e451a275629b346b1e636458361682ce7750a9721993e1"}
Oct 30 21:06:21.207183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.207161    5967 scope.go:122] "RemoveContainer" containerID="70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181"
Oct 30 21:06:21.207551 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:21.207458    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181\": container with ID starting with 70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181 not found: ID does not exist" containerID="70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181"
Oct 30 21:06:21.207551 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.207487    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181"} err="failed to get container status \"70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181\": rpc error: code = NotFound desc = could not find container \"70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181\": container with ID starting with 70ed1a4ec481a002ade93495e608ac2ad45c500000d215833eec0c6a66d4b181 not found: ID does not exist"
Oct 30 21:06:21.207551 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.207504    5967 scope.go:122] "RemoveContainer" containerID="ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c"
Oct 30 21:06:21.217055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.217038    5967 scope.go:122] "RemoveContainer" containerID="ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c"
Oct 30 21:06:21.217352 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:21.217333    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c\": container with ID starting with ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c not found: ID does not exist" containerID="ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c"
Oct 30 21:06:21.217412 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.217361    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c"} err="failed to get container status \"ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c\": rpc error: code = NotFound desc = could not find container \"ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c\": container with ID starting with ea8b8edcd247ebef464b59a6e45cc2acf823b8d0d15da11f022d0fac1587096c not found: ID does not exist"
Oct 30 21:06:21.358698 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.358606    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"]
Oct 30 21:06:21.407622 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.407587    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k"]
Oct 30 21:06:21.467929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.467898    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"]
Oct 30 21:06:21.473461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.473435    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt"]
Oct 30 21:06:21.543810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.543774    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" path="/var/lib/kubelet/pods/9918a6dc-843b-483b-ab96-c50c676c13de/volumes"
Oct 30 21:06:21.544337 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:21.544310    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" path="/var/lib/kubelet/pods/fbb207bc-4523-44b8-88af-5c038f22a2ef/volumes"
Oct 30 21:06:22.332804 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:22.332772    5967 scope.go:122] "RemoveContainer" containerID="f61280a4dcb4ed2c75f0a7e26131aa23a3aaac46f45e470781fe687e205e25d5"
Oct 30 21:06:22.344755 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:22.344713    5967 scope.go:122] "RemoveContainer" containerID="4407cdfeca1c6ceb0c3eaae049cb42e82ebcc4af0d2f93c53497c9a1674de158"
Oct 30 21:06:31.979763 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:31.979700    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:06:34.918717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:34.918683    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"]
Oct 30 21:06:34.969032 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:34.968988    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a78ac-bd4p4"]
Oct 30 21:06:35.542232 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:35.542198    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cffaad-703e-4713-a04d-2875aca681d4" path="/var/lib/kubelet/pods/90cffaad-703e-4713-a04d-2875aca681d4/volumes"
Oct 30 21:06:41.639946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:41.639901    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"]
Oct 30 21:06:41.640862 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:41.640167    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy" containerID="cri-o://e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c" gracePeriod=30
Oct 30 21:06:46.901419 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.901391    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 21:06:46.977039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977004    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977050    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977079    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977256    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977309    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977323    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:46.977366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977346    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977548 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977458    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977548 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977510    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:46.977548 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977522    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:46.977824 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977636    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:46.977899 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977851    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:46.978117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977948    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:46.978117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977963    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:46.978117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977973    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb7b62c-26d5-4657-b164-c1549cb4942d-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:46.978117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.977984    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:46.979567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.979547    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:46.979672 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.979566    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "kube-api-access-77r8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:46.979672 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.979568    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:46.979793 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:46.979779    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:47.078432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078387    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data\") pod \"4fb7b62c-26d5-4657-b164-c1549cb4942d\" (UID: \"4fb7b62c-26d5-4657-b164-c1549cb4942d\") "
Oct 30 21:06:47.078642 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078622    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data" pod "4fb7b62c-26d5-4657-b164-c1549cb4942d" (UID: "4fb7b62c-26d5-4657-b164-c1549cb4942d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:47.078856 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078835    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:47.078913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078861    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:47.078913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078874    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:47.078913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078888    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77r8x\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-kube-api-access-77r8x\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:47.078913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.078903    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb7b62c-26d5-4657-b164-c1549cb4942d-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:47.365826 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.365717    5967 generic.go:356] "Generic (PLEG): container finished" podID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerID="e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c" exitCode=0
Oct 30 21:06:47.365826 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.365789    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"
Oct 30 21:06:47.366043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.365778    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" event={"ID":"4fb7b62c-26d5-4657-b164-c1549cb4942d","Type":"ContainerDied","Data":"e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c"}
Oct 30 21:06:47.366043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.365925    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj" event={"ID":"4fb7b62c-26d5-4657-b164-c1549cb4942d","Type":"ContainerDied","Data":"f5e93cdc8416d927858df7fc59f81e25d69ae33ca372bf6eb4ab9a8aa93e9158"}
Oct 30 21:06:47.366043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.365948    5967 scope.go:122] "RemoveContainer" containerID="e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c"
Oct 30 21:06:47.377486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.377463    5967 scope.go:122] "RemoveContainer" containerID="bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"
Oct 30 21:06:47.387117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.387094    5967 scope.go:122] "RemoveContainer" containerID="e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c"
Oct 30 21:06:47.387486 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:47.387458    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c\": container with ID starting with e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c not found: ID does not exist" containerID="e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c"
Oct 30 21:06:47.387573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.387498    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c"} err="failed to get container status \"e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c\": rpc error: code = NotFound desc = could not find container \"e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c\": container with ID starting with e3b9874dff93e01bbd796d52a14d79f78a07223121fd7ca59e47167980560d4c not found: ID does not exist"
Oct 30 21:06:47.387573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.387523    5967 scope.go:122] "RemoveContainer" containerID="bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"
Oct 30 21:06:47.387858 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:06:47.387834    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f\": container with ID starting with bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f not found: ID does not exist" containerID="bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"
Oct 30 21:06:47.387964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.387863    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f"} err="failed to get container status \"bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f\": rpc error: code = NotFound desc = could not find container \"bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f\": container with ID starting with bd357796034cf5cf9c671ee435d00200730a2aacaab8a349125f955614919f1f not found: ID does not exist"
Oct 30 21:06:47.529593 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.529546    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"]
Oct 30 21:06:47.578568 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:47.578534    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj"]
Oct 30 21:06:49.549881 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:06:49.549839    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" path="/var/lib/kubelet/pods/4fb7b62c-26d5-4657-b164-c1549cb4942d/volumes"
Oct 30 21:07:01.980433 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:07:01.980356    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:07:22.405193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:22.405159    5967 scope.go:122] "RemoveContainer" containerID="1acbf2c917f9854746a5f498ab56f5a95b9abd4976105a90bede938ae5c518aa"
Oct 30 21:07:22.415077 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:22.415048    5967 scope.go:122] "RemoveContainer" containerID="e1c2644b10ea9bd763ba5f44b33fbc4634e70595b3889eaf624f4cb6387d9169"
Oct 30 21:07:22.424624 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:22.424594    5967 scope.go:122] "RemoveContainer" containerID="96d87f53e3cc9f27b2534efeff037743c20daa799fc6eaa8f9b64661c28373c0"
Oct 30 21:07:27.888830 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.888784    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"]
Oct 30 21:07:27.889502 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889480    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3005d396-c394-4ff8-8ecc-dd6d2274adda" containerName="revoke-keys"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889504    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3005d396-c394-4ff8-8ecc-dd6d2274adda" containerName="revoke-keys"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889521    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889530    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889549    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889557    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889575    5967 cpu_manager.go:414] "containerMap: removing container" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889582    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889592    5967 cpu_manager.go:414] "containerMap: removing container" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy"
Oct 30 21:07:27.889601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889600    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889616    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889625    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889636    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" containerName="payload-pre-processing"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889643    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" containerName="payload-pre-processing"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889691    5967 cpu_manager.go:414] "containerMap: removing container" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889699    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889713    5967 cpu_manager.go:414] "containerMap: removing container" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerName="payload-processing"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889720    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerName="payload-processing"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889758    5967 cpu_manager.go:414] "containerMap: removing container" podUID="dbda826e-1568-4840-a766-6d6db04d3ca8" containerName="revoke-keys"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889766    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbda826e-1568-4840-a766-6d6db04d3ca8" containerName="revoke-keys"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889887    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="29698393-dfcb-4a5c-8d99-eba4d45f1aba" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889901    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb207bc-4523-44b8-88af-5c038f22a2ef" containerName="payload-processing"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889910    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbda826e-1568-4840-a766-6d6db04d3ca8" containerName="revoke-keys"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889921    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3005d396-c394-4ff8-8ecc-dd6d2274adda" containerName="revoke-keys"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889930    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889939    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9780fc58-0b21-4bd0-81c4-ca1676765f8c" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889950    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb7b62c-26d5-4657-b164-c1549cb4942d" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889960    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="34803b24-bb46-4c76-86a9-2a7b646d9160" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889970    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e174b0ce-6120-43db-a840-bf69db117b7f" containerName="istio-proxy"
Oct 30 21:07:27.890121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.889978    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9918a6dc-843b-483b-ab96-c50c676c13de" containerName="payload-pre-processing"
Oct 30 21:07:27.895322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.895299    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:27.950706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.950670    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"]
Oct 30 21:07:27.968332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.968298    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:27.968332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.968318    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:07:27.968332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.968301    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:07:27.996691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:27.996654    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:07:28.012121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.012083    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6svp\" (UniqueName: \"kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.012318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.012266    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.012318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.012296    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.113253 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.113213    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.113253 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.113256    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.113481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.113293    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6svp\" (UniqueName: \"kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.113964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.113945    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.115845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.115819    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.179513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.179441    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6svp\" (UniqueName: \"kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp\") pod \"maas-api-revoke-sub-7237cd86-cdrrc\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.209444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.209401    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:28.586766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.586720    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"]
Oct 30 21:07:28.589310 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:07:28.589282    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92482e4b_ea84_4c9d_832c_23d893cde9c5.slice/crio-3b070971375c6038fd1bfe915981ada92a8c22f9056a0764d49abcad56717160 WatchSource:0}: Error finding container 3b070971375c6038fd1bfe915981ada92a8c22f9056a0764d49abcad56717160: Status 404 returned error can't find the container with id 3b070971375c6038fd1bfe915981ada92a8c22f9056a0764d49abcad56717160
Oct 30 21:07:28.616282 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:28.616248    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc" event={"ID":"92482e4b-ea84-4c9d-832c-23d893cde9c5","Type":"ContainerStarted","Data":"3b070971375c6038fd1bfe915981ada92a8c22f9056a0764d49abcad56717160"}
Oct 30 21:07:29.624498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:29.624459    5967 generic.go:356] "Generic (PLEG): container finished" podID="92482e4b-ea84-4c9d-832c-23d893cde9c5" containerID="69a91d4aad9ea3d5560085e2029fca96476530ea619a35f51338611e9cbb5e8e" exitCode=0
Oct 30 21:07:29.624966 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:29.624545    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc" event={"ID":"92482e4b-ea84-4c9d-832c-23d893cde9c5","Type":"ContainerDied","Data":"69a91d4aad9ea3d5560085e2029fca96476530ea619a35f51338611e9cbb5e8e"}
Oct 30 21:07:30.776233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.776200    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:30.844036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.843991    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca\") pod \"92482e4b-ea84-4c9d-832c-23d893cde9c5\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") "
Oct 30 21:07:30.844254 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.844056    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp\" (UniqueName: \"kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp\") pod \"92482e4b-ea84-4c9d-832c-23d893cde9c5\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") "
Oct 30 21:07:30.844254 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.844151    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca\") pod \"92482e4b-ea84-4c9d-832c-23d893cde9c5\" (UID: \"92482e4b-ea84-4c9d-832c-23d893cde9c5\") "
Oct 30 21:07:30.844409 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.844377    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca" pod "92482e4b-ea84-4c9d-832c-23d893cde9c5" (UID: "92482e4b-ea84-4c9d-832c-23d893cde9c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:30.844533 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.844519    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:30.846472 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.846442    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca" pod "92482e4b-ea84-4c9d-832c-23d893cde9c5" (UID: "92482e4b-ea84-4c9d-832c-23d893cde9c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:30.846641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.846518    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp" pod "92482e4b-ea84-4c9d-832c-23d893cde9c5" (UID: "92482e4b-ea84-4c9d-832c-23d893cde9c5"). InnerVolumeSpecName "kube-api-access-w6svp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:30.945695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.945602    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6svp\" (UniqueName: \"kubernetes.io/projected/92482e4b-ea84-4c9d-832c-23d893cde9c5-kube-api-access-w6svp\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:30.945695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:30.945635    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92482e4b-ea84-4c9d-832c-23d893cde9c5-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:31.638914 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:31.638884    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"
Oct 30 21:07:31.639087 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:31.638918    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc" event={"ID":"92482e4b-ea84-4c9d-832c-23d893cde9c5","Type":"ContainerDied","Data":"3b070971375c6038fd1bfe915981ada92a8c22f9056a0764d49abcad56717160"}
Oct 30 21:07:31.639087 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:31.638971    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b070971375c6038fd1bfe915981ada92a8c22f9056a0764d49abcad56717160"
Oct 30 21:07:31.980799 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:07:31.980725    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:07:39.841298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:39.841253    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"]
Oct 30 21:07:40.802877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.802838    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"]
Oct 30 21:07:40.803335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.803322    5967 cpu_manager.go:414] "containerMap: removing container" podUID="92482e4b-ea84-4c9d-832c-23d893cde9c5" containerName="revoke-keys"
Oct 30 21:07:40.803378 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.803336    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92482e4b-ea84-4c9d-832c-23d893cde9c5" containerName="revoke-keys"
Oct 30 21:07:40.803474 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.803424    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="92482e4b-ea84-4c9d-832c-23d893cde9c5" containerName="revoke-keys"
Oct 30 21:07:40.806890 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.806870    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:40.861157 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.861102    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:07:40.861157 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.861102    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:40.862273 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.861245    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:07:40.864172 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.864141    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"]
Oct 30 21:07:40.910359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.910320    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:07:40.948233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.948188    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:40.948233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.948235    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75d9f\" (UniqueName: \"kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:40.948516 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:40.948388    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.049261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.049204    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.049481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.049341    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.049481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.049360    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75d9f\" (UniqueName: \"kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.049994 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.049969    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.052207 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.052180    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.106078 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.105995    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75d9f\" (UniqueName: \"kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f\") pod \"maas-api-revoke-sub-79d3b76c-m9hrl\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.120353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.120318    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:41.499902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.499874    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"]
Oct 30 21:07:41.501862 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:07:41.501824    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983037f4_4245_4ee9_bbc4_64e7f0a01cb1.slice/crio-57e41d3f7cb9d3c8e3b6d17253fced20ca5cb9b9b7462493a2db1e55a418e44e WatchSource:0}: Error finding container 57e41d3f7cb9d3c8e3b6d17253fced20ca5cb9b9b7462493a2db1e55a418e44e: Status 404 returned error can't find the container with id 57e41d3f7cb9d3c8e3b6d17253fced20ca5cb9b9b7462493a2db1e55a418e44e
Oct 30 21:07:41.707136 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:41.707099    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl" event={"ID":"983037f4-4245-4ee9-bbc4-64e7f0a01cb1","Type":"ContainerStarted","Data":"57e41d3f7cb9d3c8e3b6d17253fced20ca5cb9b9b7462493a2db1e55a418e44e"}
Oct 30 21:07:42.714521 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:42.714497    5967 generic.go:356] "Generic (PLEG): container finished" podID="983037f4-4245-4ee9-bbc4-64e7f0a01cb1" containerID="34e4e84e8fbec6436da94ac43d30afb6bca5110867f48f65120851eb3f02705c" exitCode=0
Oct 30 21:07:42.714950 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:42.714567    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl" event={"ID":"983037f4-4245-4ee9-bbc4-64e7f0a01cb1","Type":"ContainerDied","Data":"34e4e84e8fbec6436da94ac43d30afb6bca5110867f48f65120851eb3f02705c"}
Oct 30 21:07:43.866105 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.866078    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:43.983189 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.983098    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca\") pod \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") "
Oct 30 21:07:43.983437 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.983213    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca\") pod \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") "
Oct 30 21:07:43.983437 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.983363    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f\" (UniqueName: \"kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f\") pod \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\" (UID: \"983037f4-4245-4ee9-bbc4-64e7f0a01cb1\") "
Oct 30 21:07:43.983569 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.983506    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca" pod "983037f4-4245-4ee9-bbc4-64e7f0a01cb1" (UID: "983037f4-4245-4ee9-bbc4-64e7f0a01cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:43.983775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.983719    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:43.985857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.985830    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca" pod "983037f4-4245-4ee9-bbc4-64e7f0a01cb1" (UID: "983037f4-4245-4ee9-bbc4-64e7f0a01cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:43.985997 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:43.985874    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f" pod "983037f4-4245-4ee9-bbc4-64e7f0a01cb1" (UID: "983037f4-4245-4ee9-bbc4-64e7f0a01cb1"). InnerVolumeSpecName "kube-api-access-75d9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:44.084528 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:44.084495    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75d9f\" (UniqueName: \"kubernetes.io/projected/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-kube-api-access-75d9f\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:44.084528 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:44.084523    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983037f4-4245-4ee9-bbc4-64e7f0a01cb1-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:44.729153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:44.729123    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"
Oct 30 21:07:44.729326 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:44.729160    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl" event={"ID":"983037f4-4245-4ee9-bbc4-64e7f0a01cb1","Type":"ContainerDied","Data":"57e41d3f7cb9d3c8e3b6d17253fced20ca5cb9b9b7462493a2db1e55a418e44e"}
Oct 30 21:07:44.729326 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:44.729216    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57e41d3f7cb9d3c8e3b6d17253fced20ca5cb9b9b7462493a2db1e55a418e44e"
Oct 30 21:07:54.862106 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:54.862057    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk" podUID="58d3d030-84f5-4478-80eb-54a7e5c09fb7" containerName="main" containerID="cri-o://438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260" gracePeriod=45
Oct 30 21:07:55.124115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.124050    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:07:55.199499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.199461    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm\") pod \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") "
Oct 30 21:07:55.199712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.199521    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw\" (UniqueName: \"kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw\") pod \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") "
Oct 30 21:07:55.199712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.199619    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache\") pod \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") "
Oct 30 21:07:55.199712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.199664    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs\") pod \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") "
Oct 30 21:07:55.199965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.199936    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache" pod "58d3d030-84f5-4478-80eb-54a7e5c09fb7" (UID: "58d3d030-84f5-4478-80eb-54a7e5c09fb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:55.200128 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.200110    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:55.201933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.201905    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm" pod "58d3d030-84f5-4478-80eb-54a7e5c09fb7" (UID: "58d3d030-84f5-4478-80eb-54a7e5c09fb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:55.202056 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.201930    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs" pod "58d3d030-84f5-4478-80eb-54a7e5c09fb7" (UID: "58d3d030-84f5-4478-80eb-54a7e5c09fb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:55.202056 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.202009    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw" pod "58d3d030-84f5-4478-80eb-54a7e5c09fb7" (UID: "58d3d030-84f5-4478-80eb-54a7e5c09fb7"). InnerVolumeSpecName "kube-api-access-8czhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:55.300553 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300518    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home\") pod \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") "
Oct 30 21:07:55.300813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300584    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir\") pod \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\" (UID: \"58d3d030-84f5-4478-80eb-54a7e5c09fb7\") "
Oct 30 21:07:55.300882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300871    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:55.300924 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300885    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:55.300924 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300896    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8czhw\" (UniqueName: \"kubernetes.io/projected/58d3d030-84f5-4478-80eb-54a7e5c09fb7-kube-api-access-8czhw\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:55.300924 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300899    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home" pod "58d3d030-84f5-4478-80eb-54a7e5c09fb7" (UID: "58d3d030-84f5-4478-80eb-54a7e5c09fb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:55.301026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.300895    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir" pod "58d3d030-84f5-4478-80eb-54a7e5c09fb7" (UID: "58d3d030-84f5-4478-80eb-54a7e5c09fb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:55.401446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.401351    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:55.401446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.401392    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d3d030-84f5-4478-80eb-54a7e5c09fb7-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:55.793463 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.793431    5967 generic.go:356] "Generic (PLEG): container finished" podID="58d3d030-84f5-4478-80eb-54a7e5c09fb7" containerID="438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260" exitCode=0
Oct 30 21:07:55.793655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.793470    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"
Oct 30 21:07:55.793655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.793515    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk" event={"ID":"58d3d030-84f5-4478-80eb-54a7e5c09fb7","Type":"ContainerDied","Data":"438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260"}
Oct 30 21:07:55.793655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.793562    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk" event={"ID":"58d3d030-84f5-4478-80eb-54a7e5c09fb7","Type":"ContainerDied","Data":"cb8e118934f23ed9104f55729ace4cb958e07524741d83c3ec5291f4433b1ba7"}
Oct 30 21:07:55.793655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.793581    5967 scope.go:122] "RemoveContainer" containerID="438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260"
Oct 30 21:07:55.803418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.803398    5967 scope.go:122] "RemoveContainer" containerID="438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260"
Oct 30 21:07:55.803704 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:07:55.803683    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260\": container with ID starting with 438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260 not found: ID does not exist" containerID="438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260"
Oct 30 21:07:55.803787 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.803714    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260"} err="failed to get container status \"438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260\": rpc error: code = NotFound desc = could not find container \"438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260\": container with ID starting with 438deedb36d5207a69ac59a78de2008cc33b4aa5a9ef79862a39a3d6dfbfd260 not found: ID does not exist"
Oct 30 21:07:55.955499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.955463    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"]
Oct 30 21:07:55.958688 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:55.958655    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2fa881/test-model-2fa881-kserve-5cfdbfd5c5-vfrsk"]
Oct 30 21:07:57.542058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:07:57.542018    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d3d030-84f5-4478-80eb-54a7e5c09fb7" path="/var/lib/kubelet/pods/58d3d030-84f5-4478-80eb-54a7e5c09fb7/volumes"
Oct 30 21:08:01.980084 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:08:01.980042    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:08:08.566517 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.566476    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"]
Oct 30 21:08:08.567229 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.567209    5967 cpu_manager.go:414] "containerMap: removing container" podUID="58d3d030-84f5-4478-80eb-54a7e5c09fb7" containerName="main"
Oct 30 21:08:08.567286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.567233    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d3d030-84f5-4478-80eb-54a7e5c09fb7" containerName="main"
Oct 30 21:08:08.567329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.567306    5967 cpu_manager.go:414] "containerMap: removing container" podUID="983037f4-4245-4ee9-bbc4-64e7f0a01cb1" containerName="revoke-keys"
Oct 30 21:08:08.567329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.567315    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983037f4-4245-4ee9-bbc4-64e7f0a01cb1" containerName="revoke-keys"
Oct 30 21:08:08.567440 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.567429    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="983037f4-4245-4ee9-bbc4-64e7f0a01cb1" containerName="revoke-keys"
Oct 30 21:08:08.567477 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.567447    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d3d030-84f5-4478-80eb-54a7e5c09fb7" containerName="main"
Oct 30 21:08:08.572175 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.572151    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.625694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.625666    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:08.625954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.625667    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:08.625954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.625693    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:08.626443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.626420    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"]
Oct 30 21:08:08.632295 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.632268    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.632417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.632309    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9hn\" (UniqueName: \"kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.632555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.632527    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.672504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.672476    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:08:08.733541 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.733506    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.733849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.733562    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.733849 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.733589    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9hn\" (UniqueName: \"kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.734347 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.734321    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.736155 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.736132    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.791380 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.791347    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9hn\" (UniqueName: \"kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn\") pod \"maas-api-revoke-keys-e2e-inf-a-2fa881-77df2\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:08.885655 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:08.885581    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:09.259880 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:09.259851    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"]
Oct 30 21:08:09.262151 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:08:09.262122    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e5c7ae_8b05_41df_8885_ee161ccbaf8d.slice/crio-3a8d3394e4506ad2f9ac3177e13de5287a59c9f5bdab0e3fd078c31a5030a7fb WatchSource:0}: Error finding container 3a8d3394e4506ad2f9ac3177e13de5287a59c9f5bdab0e3fd078c31a5030a7fb: Status 404 returned error can't find the container with id 3a8d3394e4506ad2f9ac3177e13de5287a59c9f5bdab0e3fd078c31a5030a7fb
Oct 30 21:08:09.884311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:09.884224    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2" event={"ID":"84e5c7ae-8b05-41df-8885-ee161ccbaf8d","Type":"ContainerStarted","Data":"2ad3a44c5cff6b75ce4d2e184dbadab263f6f01ef1dca52b67400089c52ea799"}
Oct 30 21:08:09.884311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:09.884279    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2" event={"ID":"84e5c7ae-8b05-41df-8885-ee161ccbaf8d","Type":"ContainerStarted","Data":"3a8d3394e4506ad2f9ac3177e13de5287a59c9f5bdab0e3fd078c31a5030a7fb"}
Oct 30 21:08:09.999664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:09.999610    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2" podStartSLOduration=1.9995839549999999 podStartE2EDuration="1.999583955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:09.996735908 +0000 UTC m=+2870.597537863" watchObservedRunningTime="2026-10-30 21:08:09.999583955 +0000 UTC m=+2870.600385910"
Oct 30 21:08:10.891580 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:10.891544    5967 generic.go:356] "Generic (PLEG): container finished" podID="84e5c7ae-8b05-41df-8885-ee161ccbaf8d" containerID="2ad3a44c5cff6b75ce4d2e184dbadab263f6f01ef1dca52b67400089c52ea799" exitCode=0
Oct 30 21:08:10.892044 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:10.891638    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2" event={"ID":"84e5c7ae-8b05-41df-8885-ee161ccbaf8d","Type":"ContainerDied","Data":"2ad3a44c5cff6b75ce4d2e184dbadab263f6f01ef1dca52b67400089c52ea799"}
Oct 30 21:08:12.038040 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.038007    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:12.168427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.168350    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca\") pod \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") "
Oct 30 21:08:12.168427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.168406    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn\" (UniqueName: \"kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn\") pod \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") "
Oct 30 21:08:12.168635 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.168579    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca\") pod \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\" (UID: \"84e5c7ae-8b05-41df-8885-ee161ccbaf8d\") "
Oct 30 21:08:12.168702 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.168686    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca" pod "84e5c7ae-8b05-41df-8885-ee161ccbaf8d" (UID: "84e5c7ae-8b05-41df-8885-ee161ccbaf8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:12.169094 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.169069    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:12.170692 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.170667    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca" pod "84e5c7ae-8b05-41df-8885-ee161ccbaf8d" (UID: "84e5c7ae-8b05-41df-8885-ee161ccbaf8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:12.170859 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.170802    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn" pod "84e5c7ae-8b05-41df-8885-ee161ccbaf8d" (UID: "84e5c7ae-8b05-41df-8885-ee161ccbaf8d"). InnerVolumeSpecName "kube-api-access-zq9hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:12.270521 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.270467    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:12.270521 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.270512    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9hn\" (UniqueName: \"kubernetes.io/projected/84e5c7ae-8b05-41df-8885-ee161ccbaf8d-kube-api-access-zq9hn\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:12.905461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.905429    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"
Oct 30 21:08:12.905461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.905447    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2" event={"ID":"84e5c7ae-8b05-41df-8885-ee161ccbaf8d","Type":"ContainerDied","Data":"3a8d3394e4506ad2f9ac3177e13de5287a59c9f5bdab0e3fd078c31a5030a7fb"}
Oct 30 21:08:12.905735 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:12.905490    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8d3394e4506ad2f9ac3177e13de5287a59c9f5bdab0e3fd078c31a5030a7fb"
Oct 30 21:08:20.069055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:20.069021    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"]
Oct 30 21:08:20.172794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:20.172761    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"]
Oct 30 21:08:22.465319 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:22.465289    5967 scope.go:122] "RemoveContainer" containerID="ebd885e43ee9fd595c778f3605de05cb133348284030aa956919a9d6baa8dbe9"
Oct 30 21:08:22.475022 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:22.474997    5967 scope.go:122] "RemoveContainer" containerID="33b713c3a03ca8732a3813d25e4a5078f4c02f0b2058931c1d336b86c2b6fe6e"
Oct 30 21:08:25.069715 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.069676    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6" podUID="6d20da6c-ea9e-4bf8-9e7d-228838469418" containerName="payload-processing" containerID="cri-o://1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d" gracePeriod=25
Oct 30 21:08:25.173809 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.173721    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerName="payload-pre-processing" containerID="cri-o://e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c" gracePeriod=25
Oct 30 21:08:25.324368 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.324310    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:08:25.502507 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.502468    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj\" (UniqueName: \"kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj\") pod \"6d20da6c-ea9e-4bf8-9e7d-228838469418\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") "
Oct 30 21:08:25.502711 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.502595    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume\") pod \"6d20da6c-ea9e-4bf8-9e7d-228838469418\" (UID: \"6d20da6c-ea9e-4bf8-9e7d-228838469418\") "
Oct 30 21:08:25.503051 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.503023    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume" pod "6d20da6c-ea9e-4bf8-9e7d-228838469418" (UID: "6d20da6c-ea9e-4bf8-9e7d-228838469418"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:25.504794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.504769    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj" pod "6d20da6c-ea9e-4bf8-9e7d-228838469418" (UID: "6d20da6c-ea9e-4bf8-9e7d-228838469418"). InnerVolumeSpecName "kube-api-access-fxmsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:25.604030 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.603936    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxmsj\" (UniqueName: \"kubernetes.io/projected/6d20da6c-ea9e-4bf8-9e7d-228838469418-kube-api-access-fxmsj\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:25.604030 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.603970    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d20da6c-ea9e-4bf8-9e7d-228838469418-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:25.982963 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.982927    5967 generic.go:356] "Generic (PLEG): container finished" podID="6d20da6c-ea9e-4bf8-9e7d-228838469418" containerID="1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d" exitCode=0
Oct 30 21:08:25.983236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.982982    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"
Oct 30 21:08:25.983236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.983005    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6" event={"ID":"6d20da6c-ea9e-4bf8-9e7d-228838469418","Type":"ContainerDied","Data":"1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d"}
Oct 30 21:08:25.983236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.983072    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6" event={"ID":"6d20da6c-ea9e-4bf8-9e7d-228838469418","Type":"ContainerDied","Data":"6b0bd3b6cbddb376ce089b4aace6e5f12aac64a35e2b9bed8d98d9eca3812483"}
Oct 30 21:08:25.983236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.983094    5967 scope.go:122] "RemoveContainer" containerID="1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d"
Oct 30 21:08:25.995847 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.995823    5967 scope.go:122] "RemoveContainer" containerID="1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d"
Oct 30 21:08:25.996191 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:08:25.996169    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d\": container with ID starting with 1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d not found: ID does not exist" containerID="1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d"
Oct 30 21:08:25.996248 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:25.996201    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d"} err="failed to get container status \"1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d\": rpc error: code = NotFound desc = could not find container \"1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d\": container with ID starting with 1607a84f5d27afa446e90150aa6b85c0dce10dfdba495ecdb9c5198b9e98c84d not found: ID does not exist"
Oct 30 21:08:26.144841 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:26.144796    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"]
Oct 30 21:08:26.194139 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:26.194105    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6"]
Oct 30 21:08:27.541233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:27.541202    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d20da6c-ea9e-4bf8-9e7d-228838469418" path="/var/lib/kubelet/pods/6d20da6c-ea9e-4bf8-9e7d-228838469418/volumes"
Oct 30 21:08:31.980843 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:08:31.980723    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:08:32.030090 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:32.030047    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:08:33.048213 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:33.048178    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"]
Oct 30 21:08:33.097586 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:33.097551    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e3393-kj5qk"]
Oct 30 21:08:33.541747 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:33.541713    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02ebacf-ac9c-4d7e-841e-81d36d8631d1" path="/var/lib/kubelet/pods/f02ebacf-ac9c-4d7e-841e-81d36d8631d1/volumes"
Oct 30 21:08:39.385906 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:39.385871    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"]
Oct 30 21:08:39.435191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:39.435156    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2fa881-77df2"]
Oct 30 21:08:39.541972 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:39.541939    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e5c7ae-8b05-41df-8885-ee161ccbaf8d" path="/var/lib/kubelet/pods/84e5c7ae-8b05-41df-8885-ee161ccbaf8d/volumes"
Oct 30 21:08:42.029998 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:42.029896    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:08:45.419322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.419294    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:08:45.498862 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.498831    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume\") pod \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") "
Oct 30 21:08:45.499082 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.498897    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5\" (UniqueName: \"kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5\") pod \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\" (UID: \"9047596e-8fb5-479b-83c0-2a7b83bfb82d\") "
Oct 30 21:08:45.499265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.499241    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume" pod "9047596e-8fb5-479b-83c0-2a7b83bfb82d" (UID: "9047596e-8fb5-479b-83c0-2a7b83bfb82d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:45.501127 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.501106    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5" pod "9047596e-8fb5-479b-83c0-2a7b83bfb82d" (UID: "9047596e-8fb5-479b-83c0-2a7b83bfb82d"). InnerVolumeSpecName "kube-api-access-9wqn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:45.600177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.600135    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9047596e-8fb5-479b-83c0-2a7b83bfb82d-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:45.600177 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.600164    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqn5\" (UniqueName: \"kubernetes.io/projected/9047596e-8fb5-479b-83c0-2a7b83bfb82d-kube-api-access-9wqn5\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:45.856042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.855950    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"]
Oct 30 21:08:45.856224 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:45.856200    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy" containerID="cri-o://48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c" gracePeriod=30
Oct 30 21:08:46.045933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.045896    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"]
Oct 30 21:08:46.048607 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.048580    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79127023-n8s5m"]
Oct 30 21:08:46.112405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.112310    5967 generic.go:356] "Generic (PLEG): container finished" podID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerID="e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c" exitCode=0
Oct 30 21:08:46.112405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.112362    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"
Oct 30 21:08:46.112405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.112393    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" event={"ID":"9047596e-8fb5-479b-83c0-2a7b83bfb82d","Type":"ContainerDied","Data":"e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c"}
Oct 30 21:08:46.112674 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.112449    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl" event={"ID":"9047596e-8fb5-479b-83c0-2a7b83bfb82d","Type":"ContainerDied","Data":"4556e6d96dfacbda7c2506a4843c86266108f9c5e1a48aed7af882602ce95a56"}
Oct 30 21:08:46.112674 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.112479    5967 scope.go:122] "RemoveContainer" containerID="e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c"
Oct 30 21:08:46.121870 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.121845    5967 scope.go:122] "RemoveContainer" containerID="e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c"
Oct 30 21:08:46.122157 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:08:46.122139    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c\": container with ID starting with e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c not found: ID does not exist" containerID="e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c"
Oct 30 21:08:46.122206 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.122169    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c"} err="failed to get container status \"e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c\": rpc error: code = NotFound desc = could not find container \"e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c\": container with ID starting with e656be53b2fdad99fd763c16faeb42028e7ed3fd7aaac6d8ff5c1516a8d2c85c not found: ID does not exist"
Oct 30 21:08:46.277218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.277174    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"]
Oct 30 21:08:46.281648 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.281618    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl"]
Oct 30 21:08:46.743322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743291    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"]
Oct 30 21:08:46.743832 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743812    5967 cpu_manager.go:414] "containerMap: removing container" podUID="6d20da6c-ea9e-4bf8-9e7d-228838469418" containerName="payload-processing"
Oct 30 21:08:46.743832 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743826    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d20da6c-ea9e-4bf8-9e7d-228838469418" containerName="payload-processing"
Oct 30 21:08:46.743947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743846    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerName="payload-pre-processing"
Oct 30 21:08:46.743947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743851    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerName="payload-pre-processing"
Oct 30 21:08:46.743947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743860    5967 cpu_manager.go:414] "containerMap: removing container" podUID="84e5c7ae-8b05-41df-8885-ee161ccbaf8d" containerName="revoke-keys"
Oct 30 21:08:46.743947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743867    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e5c7ae-8b05-41df-8885-ee161ccbaf8d" containerName="revoke-keys"
Oct 30 21:08:46.743947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743942    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d20da6c-ea9e-4bf8-9e7d-228838469418" containerName="payload-processing"
Oct 30 21:08:46.744131 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743951    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" containerName="payload-pre-processing"
Oct 30 21:08:46.744131 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.743958    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e5c7ae-8b05-41df-8885-ee161ccbaf8d" containerName="revoke-keys"
Oct 30 21:08:46.747170 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.747155    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.801473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.801409    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:46.801473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.801411    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:46.801473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.801460    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:46.801851 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.801524    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:08:46.804560 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.804523    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"]
Oct 30 21:08:46.812125 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.812093    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.812276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.812150    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.812276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.812256    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lp7j\" (UniqueName: \"kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.913252 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.913222    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.913433 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.913281    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.913433 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.913314    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lp7j\" (UniqueName: \"kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.913894 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.913858    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.915598 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.915574    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:46.969732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:46.969697    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lp7j\" (UniqueName: \"kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j\") pod \"maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:47.060532 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:47.060440    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:47.434964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:47.434934    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"]
Oct 30 21:08:47.437664 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:08:47.437637    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7217d8_94e2_4799_b5cb_089adf1eaaa1.slice/crio-0a3f383887f9abe42e5adcf9b23ac4b739e1f397546cdb07ab81525dc47bfb93 WatchSource:0}: Error finding container 0a3f383887f9abe42e5adcf9b23ac4b739e1f397546cdb07ab81525dc47bfb93: Status 404 returned error can't find the container with id 0a3f383887f9abe42e5adcf9b23ac4b739e1f397546cdb07ab81525dc47bfb93
Oct 30 21:08:47.542463 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:47.542426    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b07256-7360-46c1-9e40-ac72b54e3d38" path="/var/lib/kubelet/pods/22b07256-7360-46c1-9e40-ac72b54e3d38/volumes"
Oct 30 21:08:47.542867 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:47.542853    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9047596e-8fb5-479b-83c0-2a7b83bfb82d" path="/var/lib/kubelet/pods/9047596e-8fb5-479b-83c0-2a7b83bfb82d/volumes"
Oct 30 21:08:48.145842 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:48.145802    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs" event={"ID":"1f7217d8-94e2-4799-b5cb-089adf1eaaa1","Type":"ContainerStarted","Data":"92075d8140db68de9165c1c2d470923306f626886c600ac046c8faa0e5bb86a1"}
Oct 30 21:08:48.146426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:48.145867    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs" event={"ID":"1f7217d8-94e2-4799-b5cb-089adf1eaaa1","Type":"ContainerStarted","Data":"0a3f383887f9abe42e5adcf9b23ac4b739e1f397546cdb07ab81525dc47bfb93"}
Oct 30 21:08:48.261973 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:48.261908    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs" podStartSLOduration=2.261888596 podStartE2EDuration="2.261888596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:48.261114982 +0000 UTC m=+2908.861916945" watchObservedRunningTime="2026-10-30 21:08:48.261888596 +0000 UTC m=+2908.862690556"
Oct 30 21:08:49.153893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:49.153860    5967 generic.go:356] "Generic (PLEG): container finished" podID="1f7217d8-94e2-4799-b5cb-089adf1eaaa1" containerID="92075d8140db68de9165c1c2d470923306f626886c600ac046c8faa0e5bb86a1" exitCode=0
Oct 30 21:08:49.154332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:49.153929    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs" event={"ID":"1f7217d8-94e2-4799-b5cb-089adf1eaaa1","Type":"ContainerDied","Data":"92075d8140db68de9165c1c2d470923306f626886c600ac046c8faa0e5bb86a1"}
Oct 30 21:08:50.301978 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.301954    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:50.347505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.347477    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca\") pod \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") "
Oct 30 21:08:50.347691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.347580    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca\") pod \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") "
Oct 30 21:08:50.347691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.347658    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j\" (UniqueName: \"kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j\") pod \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\" (UID: \"1f7217d8-94e2-4799-b5cb-089adf1eaaa1\") "
Oct 30 21:08:50.347868 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.347850    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca" pod "1f7217d8-94e2-4799-b5cb-089adf1eaaa1" (UID: "1f7217d8-94e2-4799-b5cb-089adf1eaaa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:50.348150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.348129    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:50.349961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.349938    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca" pod "1f7217d8-94e2-4799-b5cb-089adf1eaaa1" (UID: "1f7217d8-94e2-4799-b5cb-089adf1eaaa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:50.350071 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.349994    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j" pod "1f7217d8-94e2-4799-b5cb-089adf1eaaa1" (UID: "1f7217d8-94e2-4799-b5cb-089adf1eaaa1"). InnerVolumeSpecName "kube-api-access-4lp7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:50.449285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.449213    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:50.449285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:50.449240    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lp7j\" (UniqueName: \"kubernetes.io/projected/1f7217d8-94e2-4799-b5cb-089adf1eaaa1-kube-api-access-4lp7j\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.098542 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.098519    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:08:51.156473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156396    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156489    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156519    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156555    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156632 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156618    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156653    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156716    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156813    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:51.156929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156816    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:51.156929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156824    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.156929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.156880    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo\") pod \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\" (UID: \"9c9a8bbd-bb10-464a-a543-6ebc193166a0\") "
Oct 30 21:08:51.157250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157019    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:51.157250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157039    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:51.157250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157123    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:51.157429 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157369    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.157429 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157388    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.157429 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157397    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.157429 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157404    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.157429 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.157416    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.159183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.159162    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:51.159275 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.159188    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "kube-api-access-rdvp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:51.159438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.159425    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:51.159477 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.159430    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo" pod "9c9a8bbd-bb10-464a-a543-6ebc193166a0" (UID: "9c9a8bbd-bb10-464a-a543-6ebc193166a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:08:51.167408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.167386    5967 generic.go:356] "Generic (PLEG): container finished" podID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerID="48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c" exitCode=0
Oct 30 21:08:51.167493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.167439    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"
Oct 30 21:08:51.167493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.167465    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" event={"ID":"9c9a8bbd-bb10-464a-a543-6ebc193166a0","Type":"ContainerDied","Data":"48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c"}
Oct 30 21:08:51.167599 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.167518    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj" event={"ID":"9c9a8bbd-bb10-464a-a543-6ebc193166a0","Type":"ContainerDied","Data":"3f10c1713c87d291ce66f9f2aea7d71f1f1ab27b9038934ad55c573fce179eee"}
Oct 30 21:08:51.167599 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.167543    5967 scope.go:122] "RemoveContainer" containerID="48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c"
Oct 30 21:08:51.169087 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.169067    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"
Oct 30 21:08:51.169190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.169131    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs" event={"ID":"1f7217d8-94e2-4799-b5cb-089adf1eaaa1","Type":"ContainerDied","Data":"0a3f383887f9abe42e5adcf9b23ac4b739e1f397546cdb07ab81525dc47bfb93"}
Oct 30 21:08:51.169190 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.169178    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a3f383887f9abe42e5adcf9b23ac4b739e1f397546cdb07ab81525dc47bfb93"
Oct 30 21:08:51.178914 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.178891    5967 scope.go:122] "RemoveContainer" containerID="e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"
Oct 30 21:08:51.187885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.187865    5967 scope.go:122] "RemoveContainer" containerID="48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c"
Oct 30 21:08:51.188149 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:08:51.188135    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c\": container with ID starting with 48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c not found: ID does not exist" containerID="48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c"
Oct 30 21:08:51.188199 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.188159    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c"} err="failed to get container status \"48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c\": rpc error: code = NotFound desc = could not find container \"48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c\": container with ID starting with 48ed82535dd4b350b73617789e2ffc66236062ada2134e18caa8422b1d11412c not found: ID does not exist"
Oct 30 21:08:51.188199 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.188178    5967 scope.go:122] "RemoveContainer" containerID="e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"
Oct 30 21:08:51.188416 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:08:51.188402    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4\": container with ID starting with e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4 not found: ID does not exist" containerID="e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"
Oct 30 21:08:51.188465 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.188417    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4"} err="failed to get container status \"e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4\": rpc error: code = NotFound desc = could not find container \"e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4\": container with ID starting with e478e8f198c19f1dd2e0490c4637ee4deec773845f263ad7dee0471f4cc92ed4 not found: ID does not exist"
Oct 30 21:08:51.258378 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.258348    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvp7\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-kube-api-access-rdvp7\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.258378 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.258376    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.258378 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.258387    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.258628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.258396    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9a8bbd-bb10-464a-a543-6ebc193166a0-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:51.392012 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.391952    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"]
Oct 30 21:08:51.394799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.394774    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj"]
Oct 30 21:08:51.542052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:51.542020    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" path="/var/lib/kubelet/pods/9c9a8bbd-bb10-464a-a543-6ebc193166a0/volumes"
Oct 30 21:08:52.066525 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:52.066484    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"]
Oct 30 21:08:52.068443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:52.068409    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"]
Oct 30 21:08:52.116762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:52.116661    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c55c59-kzhvv"]
Oct 30 21:08:52.118308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:52.118287    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d7658e-cvvpg"]
Oct 30 21:08:53.542408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:53.542375    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047724ee-8819-434f-8a6d-729958365e61" path="/var/lib/kubelet/pods/047724ee-8819-434f-8a6d-729958365e61/volumes"
Oct 30 21:08:53.542798 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:08:53.542756    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870e547d-4408-4af0-9375-ebd97e1c4ba4" path="/var/lib/kubelet/pods/870e547d-4408-4af0-9375-ebd97e1c4ba4/volumes"
Oct 30 21:09:01.981250 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:09:01.981210    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:09:09.568889 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:09.568859    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"]
Oct 30 21:09:09.570390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:09.570368    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a064b4-s2vgs"]
Oct 30 21:09:11.542143 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:11.542110    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7217d8-94e2-4799-b5cb-089adf1eaaa1" path="/var/lib/kubelet/pods/1f7217d8-94e2-4799-b5cb-089adf1eaaa1/volumes"
Oct 30 21:09:16.572838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:16.572804    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"]
Oct 30 21:09:16.573374 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:16.573020    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy" containerID="cri-o://6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df" gracePeriod=30
Oct 30 21:09:17.734982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.734946    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"]
Oct 30 21:09:17.735412 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735399    5967 cpu_manager.go:414] "containerMap: removing container" podUID="1f7217d8-94e2-4799-b5cb-089adf1eaaa1" containerName="revoke-keys"
Oct 30 21:09:17.735457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735413    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7217d8-94e2-4799-b5cb-089adf1eaaa1" containerName="revoke-keys"
Oct 30 21:09:17.735457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735423    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy"
Oct 30 21:09:17.735457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735429    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy"
Oct 30 21:09:17.735457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735440    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy"
Oct 30 21:09:17.735457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735446    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy"
Oct 30 21:09:17.735631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735509    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy"
Oct 30 21:09:17.735631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.735524    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7217d8-94e2-4799-b5cb-089adf1eaaa1" containerName="revoke-keys"
Oct 30 21:09:17.738316 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.738298    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.790835 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.790807    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:09:17.826569 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.826541    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.826777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.826654    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.826777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.826729    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4v9w\" (UniqueName: \"kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.840796 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.840770    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:09:17.840967 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.840885    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:09:17.842662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.842643    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"]
Oct 30 21:09:17.890954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.890930    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:09:17.927979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.927947    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4v9w\" (UniqueName: \"kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.928162 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.928053    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.928162 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.928146    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.928683 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.928663    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.930523 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.930504    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:17.990146 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:17.990077    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4v9w\" (UniqueName: \"kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w\") pod \"maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:18.051539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:18.051510    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:18.430312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:18.430288    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"]
Oct 30 21:09:18.432210 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:09:18.432182    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a7327c_7887_47aa_b9c7_2b61b49fabb1.slice/crio-6761fb733755a2dddad76c4ea574ed75717a9a2363093a51607c73a431c1fc89 WatchSource:0}: Error finding container 6761fb733755a2dddad76c4ea574ed75717a9a2363093a51607c73a431c1fc89: Status 404 returned error can't find the container with id 6761fb733755a2dddad76c4ea574ed75717a9a2363093a51607c73a431c1fc89
Oct 30 21:09:19.334659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:19.334624    5967 generic.go:356] "Generic (PLEG): container finished" podID="23a7327c-7887-47aa-b9c7-2b61b49fabb1" containerID="6340e159af254aa220ffd2c9e6536071b44785b9dd245cbd99526dbf23cf6a57" exitCode=0
Oct 30 21:09:19.335099 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:19.334712    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp" event={"ID":"23a7327c-7887-47aa-b9c7-2b61b49fabb1","Type":"ContainerDied","Data":"6340e159af254aa220ffd2c9e6536071b44785b9dd245cbd99526dbf23cf6a57"}
Oct 30 21:09:19.335099 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:19.334780    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp" event={"ID":"23a7327c-7887-47aa-b9c7-2b61b49fabb1","Type":"ContainerStarted","Data":"6761fb733755a2dddad76c4ea574ed75717a9a2363093a51607c73a431c1fc89"}
Oct 30 21:09:20.480022 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.479992    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:20.549526 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.549493    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w\" (UniqueName: \"kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w\") pod \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") "
Oct 30 21:09:20.549683 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.549560    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca\") pod \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") "
Oct 30 21:09:20.549683 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.549624    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca\") pod \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\" (UID: \"23a7327c-7887-47aa-b9c7-2b61b49fabb1\") "
Oct 30 21:09:20.549919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.549899    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca" pod "23a7327c-7887-47aa-b9c7-2b61b49fabb1" (UID: "23a7327c-7887-47aa-b9c7-2b61b49fabb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:20.551728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.551696    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w" pod "23a7327c-7887-47aa-b9c7-2b61b49fabb1" (UID: "23a7327c-7887-47aa-b9c7-2b61b49fabb1"). InnerVolumeSpecName "kube-api-access-f4v9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:20.551886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.551853    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca" pod "23a7327c-7887-47aa-b9c7-2b61b49fabb1" (UID: "23a7327c-7887-47aa-b9c7-2b61b49fabb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:20.650628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.650543    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:20.650628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.650574    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4v9w\" (UniqueName: \"kubernetes.io/projected/23a7327c-7887-47aa-b9c7-2b61b49fabb1-kube-api-access-f4v9w\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:20.650628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:20.650589    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a7327c-7887-47aa-b9c7-2b61b49fabb1-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.346900 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.346871    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"
Oct 30 21:09:21.347101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.346902    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp" event={"ID":"23a7327c-7887-47aa-b9c7-2b61b49fabb1","Type":"ContainerDied","Data":"6761fb733755a2dddad76c4ea574ed75717a9a2363093a51607c73a431c1fc89"}
Oct 30 21:09:21.347101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.346951    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6761fb733755a2dddad76c4ea574ed75717a9a2363093a51607c73a431c1fc89"
Oct 30 21:09:21.829607 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.829584    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:09:21.860036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.859962    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860036 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860001    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860049    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860073    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860090    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860157    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860182    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860212    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860260    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token\") pod \"8128db7f-338a-4298-9ae2-92f650cbc9d6\" (UID: \"8128db7f-338a-4298-9ae2-92f650cbc9d6\") "
Oct 30 21:09:21.860566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860466    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:21.860566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860474    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:21.860776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860659    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:21.860776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860580    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:21.860776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860673    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.860776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860734    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.860966 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.860826    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:21.862417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.862387    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:09:21.862417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.862402    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "kube-api-access-rvwv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:21.862710 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.862687    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:21.862863 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.862782    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy" pod "8128db7f-338a-4298-9ae2-92f650cbc9d6" (UID: "8128db7f-338a-4298-9ae2-92f650cbc9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:21.961708 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961669    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwv2\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-kube-api-access-rvwv2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.961708 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961699    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.961708 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961713    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.962000 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961725    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.962000 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961763    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8128db7f-338a-4298-9ae2-92f650cbc9d6-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.962000 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961775    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:21.962000 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:21.961788    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8128db7f-338a-4298-9ae2-92f650cbc9d6-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:22.095800 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.095761    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"]
Oct 30 21:09:22.354017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.353986    5967 generic.go:356] "Generic (PLEG): container finished" podID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerID="6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df" exitCode=0
Oct 30 21:09:22.354210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.354042    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"
Oct 30 21:09:22.354210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.354078    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" event={"ID":"8128db7f-338a-4298-9ae2-92f650cbc9d6","Type":"ContainerDied","Data":"6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df"}
Oct 30 21:09:22.354210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.354139    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv" event={"ID":"8128db7f-338a-4298-9ae2-92f650cbc9d6","Type":"ContainerDied","Data":"6df32ebe0c0ca1a8fc78ef52aa0d70ec2e6a1fc6ec4d025ccf1248112a26ca6f"}
Oct 30 21:09:22.354210 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.354169    5967 scope.go:122] "RemoveContainer" containerID="6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df"
Oct 30 21:09:22.364706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.364674    5967 scope.go:122] "RemoveContainer" containerID="bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"
Oct 30 21:09:22.374003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.373980    5967 scope.go:122] "RemoveContainer" containerID="6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df"
Oct 30 21:09:22.374297 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:09:22.374277    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df\": container with ID starting with 6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df not found: ID does not exist" containerID="6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df"
Oct 30 21:09:22.374371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.374313    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df"} err="failed to get container status \"6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df\": rpc error: code = NotFound desc = could not find container \"6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df\": container with ID starting with 6a91a871a1ab13048cd4506351fca7215a3c95862bfbb6e6e8ef941f5aeac4df not found: ID does not exist"
Oct 30 21:09:22.374371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.374340    5967 scope.go:122] "RemoveContainer" containerID="bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"
Oct 30 21:09:22.374607 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:09:22.374589    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3\": container with ID starting with bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3 not found: ID does not exist" containerID="bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"
Oct 30 21:09:22.374651 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.374615    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3"} err="failed to get container status \"bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3\": rpc error: code = NotFound desc = could not find container \"bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3\": container with ID starting with bbf1faebea891ff3f199ac465e44f05d103e0eb22cfa6f856b32bac0e21a70b3 not found: ID does not exist"
Oct 30 21:09:22.505664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.505625    5967 scope.go:122] "RemoveContainer" containerID="7b161a46b4ebf36659f44b12504bf019686df86e53787dd9c6fb16fffcddde97"
Oct 30 21:09:22.523110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.523078    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"]
Oct 30 21:09:22.527384 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:22.527354    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv"]
Oct 30 21:09:23.542132 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:23.542099    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" path="/var/lib/kubelet/pods/8128db7f-338a-4298-9ae2-92f650cbc9d6/volumes"
Oct 30 21:09:26.049985 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:26.049947    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"]
Oct 30 21:09:26.099799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:26.099763    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"]
Oct 30 21:09:26.101251 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:26.101228    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aab730-zwjph"]
Oct 30 21:09:26.108208 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:26.108177    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c2fb2c1-frx7l"]
Oct 30 21:09:27.541918 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:27.541880    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01cc7ad9-ec4c-4926-a716-223f338516df" path="/var/lib/kubelet/pods/01cc7ad9-ec4c-4926-a716-223f338516df/volumes"
Oct 30 21:09:27.542320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:27.542280    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cbc4a55-9a57-461f-b73d-e665c51fc56a" path="/var/lib/kubelet/pods/3cbc4a55-9a57-461f-b73d-e665c51fc56a/volumes"
Oct 30 21:09:29.249908 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:29.249876    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"]
Oct 30 21:09:31.979512 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:09:31.979475    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:09:33.079717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.079686    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"]
Oct 30 21:09:33.080163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080137    5967 cpu_manager.go:414] "containerMap: removing container" podUID="23a7327c-7887-47aa-b9c7-2b61b49fabb1" containerName="revoke-keys"
Oct 30 21:09:33.080163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080151    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a7327c-7887-47aa-b9c7-2b61b49fabb1" containerName="revoke-keys"
Oct 30 21:09:33.080237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080187    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy"
Oct 30 21:09:33.080237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080192    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy"
Oct 30 21:09:33.080237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080200    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy"
Oct 30 21:09:33.080237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080205    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy"
Oct 30 21:09:33.080366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080269    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy"
Oct 30 21:09:33.080366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080279    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8128db7f-338a-4298-9ae2-92f650cbc9d6" containerName="istio-proxy"
Oct 30 21:09:33.080366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080287    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a7327c-7887-47aa-b9c7-2b61b49fabb1" containerName="revoke-keys"
Oct 30 21:09:33.080366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.080294    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9a8bbd-bb10-464a-a543-6ebc193166a0" containerName="istio-proxy"
Oct 30 21:09:33.085309 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.085289    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.137463 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.137440    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:09:33.139973 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.139952    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:09:33.167673 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.167644    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbjt\" (UniqueName: \"kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.167827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.167679    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.167873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.167850    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.187204 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.187174    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:09:33.187339 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.187214    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:09:33.190131 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.190107    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"]
Oct 30 21:09:33.268782 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.268722    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbjt\" (UniqueName: \"kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.268962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.268804    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.268962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.268944    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.269410 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.269391    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.271450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.271427    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.325454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.325416    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbjt\" (UniqueName: \"kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt\") pod \"maas-api-revoke-sub-826541f8-vdpb8\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.398075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.397994    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:33.769046 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:33.769019    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"]
Oct 30 21:09:33.771289 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:09:33.771265    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8528cca8_1a08_4d6a_9291_3cf08da5f5c0.slice/crio-7f88b497f2116b462616ddce67cf01dc416131cb44f083cf547c40b78d30fb44 WatchSource:0}: Error finding container 7f88b497f2116b462616ddce67cf01dc416131cb44f083cf547c40b78d30fb44: Status 404 returned error can't find the container with id 7f88b497f2116b462616ddce67cf01dc416131cb44f083cf547c40b78d30fb44
Oct 30 21:09:34.250239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.250168    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r" podUID="3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" containerName="payload-processing" containerID="cri-o://9675ffa245e6c8fc63d7f0a08974af3d9c35af84b40b9959836ff65f65f485c7" gracePeriod=25
Oct 30 21:09:34.425294 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.424648    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8" event={"ID":"8528cca8-1a08-4d6a-9291-3cf08da5f5c0","Type":"ContainerStarted","Data":"e4349c03dd757f2b3aa68ab6e2846a60283f5fb964c0e2c4878c3c3d4cf8fcb3"}
Oct 30 21:09:34.425294 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.424723    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8" event={"ID":"8528cca8-1a08-4d6a-9291-3cf08da5f5c0","Type":"ContainerStarted","Data":"7f88b497f2116b462616ddce67cf01dc416131cb44f083cf547c40b78d30fb44"}
Oct 30 21:09:34.426859 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.426835    5967 generic.go:356] "Generic (PLEG): container finished" podID="3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" containerID="9675ffa245e6c8fc63d7f0a08974af3d9c35af84b40b9959836ff65f65f485c7" exitCode=0
Oct 30 21:09:34.426977 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.426906    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r" event={"ID":"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6","Type":"ContainerDied","Data":"9675ffa245e6c8fc63d7f0a08974af3d9c35af84b40b9959836ff65f65f485c7"}
Oct 30 21:09:34.506604 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.506578    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 21:09:34.539990 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.539938    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8" podStartSLOduration=1.5399237270000001 podStartE2EDuration="1.539923727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:34.538032408 +0000 UTC m=+2955.138834369" watchObservedRunningTime="2026-10-30 21:09:34.539923727 +0000 UTC m=+2955.140725699"
Oct 30 21:09:34.580439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.580412    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume\") pod \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") "
Oct 30 21:09:34.580616 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.580568    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29\" (UniqueName: \"kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29\") pod \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\" (UID: \"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6\") "
Oct 30 21:09:34.580853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.580826    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume" pod "3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" (UID: "3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:34.580982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.580966    5967 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-plugins-config-volume\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:34.582875 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.582851    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29" pod "3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" (UID: "3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6"). InnerVolumeSpecName "kube-api-access-xqh29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:34.681845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:34.681817    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqh29\" (UniqueName: \"kubernetes.io/projected/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6-kube-api-access-xqh29\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:35.434461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.434423    5967 generic.go:356] "Generic (PLEG): container finished" podID="8528cca8-1a08-4d6a-9291-3cf08da5f5c0" containerID="e4349c03dd757f2b3aa68ab6e2846a60283f5fb964c0e2c4878c3c3d4cf8fcb3" exitCode=0
Oct 30 21:09:35.434962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.434515    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8" event={"ID":"8528cca8-1a08-4d6a-9291-3cf08da5f5c0","Type":"ContainerDied","Data":"e4349c03dd757f2b3aa68ab6e2846a60283f5fb964c0e2c4878c3c3d4cf8fcb3"}
Oct 30 21:09:35.435938 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.435919    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"
Oct 30 21:09:35.435938 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.435933    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r" event={"ID":"3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6","Type":"ContainerDied","Data":"752426cbb4257a56fbe2050ba48f7234c088fc1f2f22dcf2db6fa311ba06c51e"}
Oct 30 21:09:35.436097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.435982    5967 scope.go:122] "RemoveContainer" containerID="9675ffa245e6c8fc63d7f0a08974af3d9c35af84b40b9959836ff65f65f485c7"
Oct 30 21:09:35.661022 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.660988    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"]
Oct 30 21:09:35.663330 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:35.663300    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r"]
Oct 30 21:09:36.575444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.575409    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:36.703032 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.702940    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca\") pod \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") "
Oct 30 21:09:36.703219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.703046    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt\" (UniqueName: \"kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt\") pod \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") "
Oct 30 21:09:36.703219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.703122    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca\") pod \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\" (UID: \"8528cca8-1a08-4d6a-9291-3cf08da5f5c0\") "
Oct 30 21:09:36.703456 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.703431    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca" pod "8528cca8-1a08-4d6a-9291-3cf08da5f5c0" (UID: "8528cca8-1a08-4d6a-9291-3cf08da5f5c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:36.705184 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.705159    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt" pod "8528cca8-1a08-4d6a-9291-3cf08da5f5c0" (UID: "8528cca8-1a08-4d6a-9291-3cf08da5f5c0"). InnerVolumeSpecName "kube-api-access-phbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:36.705184 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.705171    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca" pod "8528cca8-1a08-4d6a-9291-3cf08da5f5c0" (UID: "8528cca8-1a08-4d6a-9291-3cf08da5f5c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:36.804502 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.804467    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:36.804502 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.804498    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbjt\" (UniqueName: \"kubernetes.io/projected/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-kube-api-access-phbjt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:36.804502 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:36.804508    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8528cca8-1a08-4d6a-9291-3cf08da5f5c0-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.116551 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.116514    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main" containerID="cri-o://55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4" gracePeriod=45
Oct 30 21:09:37.350471 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.350445    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 21:09:37.410530 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410447    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache\") pod \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") "
Oct 30 21:09:37.410530 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410507    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home\") pod \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") "
Oct 30 21:09:37.410777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410533    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm\") pod \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") "
Oct 30 21:09:37.410777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410590    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs\") pod \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") "
Oct 30 21:09:37.410777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410634    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc\" (UniqueName: \"kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc\") pod \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") "
Oct 30 21:09:37.410777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410721    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home" pod "5b0ae2fd-4e60-48d8-9318-fb494a7594e5" (UID: "5b0ae2fd-4e60-48d8-9318-fb494a7594e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.410777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410762    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache" pod "5b0ae2fd-4e60-48d8-9318-fb494a7594e5" (UID: "5b0ae2fd-4e60-48d8-9318-fb494a7594e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.411075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.410834    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir\") pod \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\" (UID: \"5b0ae2fd-4e60-48d8-9318-fb494a7594e5\") "
Oct 30 21:09:37.411141 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.411075    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir" pod "5b0ae2fd-4e60-48d8-9318-fb494a7594e5" (UID: "5b0ae2fd-4e60-48d8-9318-fb494a7594e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.411266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.411247    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.411499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.411271    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.411499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.411286    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.412975 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.412945    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm" pod "5b0ae2fd-4e60-48d8-9318-fb494a7594e5" (UID: "5b0ae2fd-4e60-48d8-9318-fb494a7594e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.413087 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.412985    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc" pod "5b0ae2fd-4e60-48d8-9318-fb494a7594e5" (UID: "5b0ae2fd-4e60-48d8-9318-fb494a7594e5"). InnerVolumeSpecName "kube-api-access-9bjcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:37.413087 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.413012    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs" pod "5b0ae2fd-4e60-48d8-9318-fb494a7594e5" (UID: "5b0ae2fd-4e60-48d8-9318-fb494a7594e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:37.450056 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.450030    5967 generic.go:356] "Generic (PLEG): container finished" podID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerID="55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4" exitCode=0
Oct 30 21:09:37.450187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.450074    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"
Oct 30 21:09:37.450187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.450112    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" event={"ID":"5b0ae2fd-4e60-48d8-9318-fb494a7594e5","Type":"ContainerDied","Data":"55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4"}
Oct 30 21:09:37.450187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.450170    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4" event={"ID":"5b0ae2fd-4e60-48d8-9318-fb494a7594e5","Type":"ContainerDied","Data":"8dfeaf85f110541ac52a2296d3ce19575f2e24f09684cdb75256e43eb4538661"}
Oct 30 21:09:37.450187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.450186    5967 scope.go:122] "RemoveContainer" containerID="55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4"
Oct 30 21:09:37.451625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.451608    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8" event={"ID":"8528cca8-1a08-4d6a-9291-3cf08da5f5c0","Type":"ContainerDied","Data":"7f88b497f2116b462616ddce67cf01dc416131cb44f083cf547c40b78d30fb44"}
Oct 30 21:09:37.451685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.451669    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f88b497f2116b462616ddce67cf01dc416131cb44f083cf547c40b78d30fb44"
Oct 30 21:09:37.451685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.451615    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"
Oct 30 21:09:37.461802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.461780    5967 scope.go:122] "RemoveContainer" containerID="55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4"
Oct 30 21:09:37.462062 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:09:37.462044    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4\": container with ID starting with 55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4 not found: ID does not exist" containerID="55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4"
Oct 30 21:09:37.462107 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.462069    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4"} err="failed to get container status \"55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4\": rpc error: code = NotFound desc = could not find container \"55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4\": container with ID starting with 55a704784ceb1e134cc5411e46c4809dae5456cb78b5fcbcb244eb1b4391cfe4 not found: ID does not exist"
Oct 30 21:09:37.512039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.512012    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.512039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.512038    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.512241 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.512048    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjcc\" (UniqueName: \"kubernetes.io/projected/5b0ae2fd-4e60-48d8-9318-fb494a7594e5-kube-api-access-9bjcc\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.542717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.542679    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" path="/var/lib/kubelet/pods/3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6/volumes"
Oct 30 21:09:37.638860 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.638824    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"]
Oct 30 21:09:37.643182 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:37.643159    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/distinct-2-w6-4ee72d-kserve-7c7474757c-57cd4"]
Oct 30 21:09:39.541018 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:39.540984    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" path="/var/lib/kubelet/pods/5b0ae2fd-4e60-48d8-9318-fb494a7594e5/volumes"
Oct 30 21:09:50.539877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:50.539845    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"]
Oct 30 21:09:50.588972 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:50.588940    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-4ee72d-n6dnp"]
Oct 30 21:09:51.541266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:09:51.541237    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a7327c-7887-47aa-b9c7-2b61b49fabb1" path="/var/lib/kubelet/pods/23a7327c-7887-47aa-b9c7-2b61b49fabb1/volumes"
Oct 30 21:10:01.979415 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:10:01.979337    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:10:05.047229 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047197    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-756d9799fc-cfpl2"]
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047658    5967 cpu_manager.go:414] "containerMap: removing container" podUID="8528cca8-1a08-4d6a-9291-3cf08da5f5c0" containerName="revoke-keys"
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047673    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8528cca8-1a08-4d6a-9291-3cf08da5f5c0" containerName="revoke-keys"
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047694    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" containerName="payload-processing"
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047700    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" containerName="payload-processing"
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047707    5967 cpu_manager.go:414] "containerMap: removing container" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main"
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047712    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main"
Oct 30 21:10:05.047808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047807    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="8528cca8-1a08-4d6a-9291-3cf08da5f5c0" containerName="revoke-keys"
Oct 30 21:10:05.048214 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047820    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3843b4e9-ba5b-4a9d-b53e-c3e9f04d44d6" containerName="payload-processing"
Oct 30 21:10:05.048214 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.047826    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0ae2fd-4e60-48d8-9318-fb494a7594e5" containerName="main"
Oct 30 21:10:05.051347 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.051327    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.080401 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.080374    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.080589 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.080411    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.080663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.080585    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp7t\" (UniqueName: \"kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.104146 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.104119    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-42mmb\""
Oct 30 21:10:05.153865 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.153835    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:10:05.153865 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.153859    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:10:05.154102 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.153879    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:10:05.155560 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.155542    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d9799fc-cfpl2"]
Oct 30 21:10:05.181640 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.181619    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.181734 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.181721    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp7t\" (UniqueName: \"kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.181817 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.181762    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.184273 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.184250    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.184383 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.184282    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.243037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.243005    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp7t\" (UniqueName: \"kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t\") pod \"maas-controller-756d9799fc-cfpl2\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") " pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.364597 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.364513    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:05.741193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.741162    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d9799fc-cfpl2"]
Oct 30 21:10:05.743767 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:10:05.743721    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4434f49_5004_40a0_9f58_b0371bbc5925.slice/crio-60923f5cd85c1d4312fd85c33253e12646c1afc1f0f18bd36fcbe39a935d4180 WatchSource:0}: Error finding container 60923f5cd85c1d4312fd85c33253e12646c1afc1f0f18bd36fcbe39a935d4180: Status 404 returned error can't find the container with id 60923f5cd85c1d4312fd85c33253e12646c1afc1f0f18bd36fcbe39a935d4180
Oct 30 21:10:05.745013 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:05.744997    5967 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:10:06.611041 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:06.611002    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-cfpl2" event={"ID":"a4434f49-5004-40a0-9f58-b0371bbc5925","Type":"ContainerStarted","Data":"60923f5cd85c1d4312fd85c33253e12646c1afc1f0f18bd36fcbe39a935d4180"}
Oct 30 21:10:08.625830 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:08.625793    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-cfpl2" event={"ID":"a4434f49-5004-40a0-9f58-b0371bbc5925","Type":"ContainerStarted","Data":"a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468"}
Oct 30 21:10:08.626244 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:08.625976    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:08.745202 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:08.745141    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-756d9799fc-cfpl2" podStartSLOduration=1.68946393 podStartE2EDuration="3.74512777s" totalImagesPullingTime="2.05566384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:08.744338511 +0000 UTC m=+2989.345140471" watchObservedRunningTime="2026-10-30 21:10:08.74512777 +0000 UTC m=+2989.345929730"
Oct 30 21:10:19.633803 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:19.633768    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:22.557130 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.557098    5967 scope.go:122] "RemoveContainer" containerID="5fa197d82c422ffbf607f6c196b81a09edb1f87a13e5e50f98f60b30111f9f7a"
Oct 30 21:10:22.566285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.566265    5967 scope.go:122] "RemoveContainer" containerID="b3e15331a8cbe8f7a6888c636e7ffb2e299002da338196f3b7852f33301c7f13"
Oct 30 21:10:22.573545 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.573518    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"]
Oct 30 21:10:22.574935 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.574913    5967 scope.go:122] "RemoveContainer" containerID="b2058628b69e8de5eaf911992fa1e115b12c30b018b751e0aa27658d97906cee"
Oct 30 21:10:22.578510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.578492    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.583585 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.583562    5967 scope.go:122] "RemoveContainer" containerID="908670df4950c37b93917bc0901abeb7610f731dc9cbdf0fc22fe2f7c3aebbf8"
Oct 30 21:10:22.593977 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.593960    5967 scope.go:122] "RemoveContainer" containerID="e5fdd7159505c438dd7952b7f4b5f1654fdc64910b363e8d2db8322e9687dd6f"
Oct 30 21:10:22.602563 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.602543    5967 scope.go:122] "RemoveContainer" containerID="84dbf435f4bac528af696d0ed753171f699c302694bd988bf4c0b8a7c6724a7a"
Oct 30 21:10:22.622984 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.622955    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"]
Oct 30 21:10:22.628879 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.628858    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.630978 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.630887    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:10:22.653007 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.652983    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653116 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653079    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653124    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653145    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653214    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653314 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653285    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653367    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653442 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653397    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fx7b\" (UniqueName: \"kubernetes.io/projected/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-kube-api-access-8fx7b\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.653556 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.653519    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.670630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.670607    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7k2d5\""
Oct 30 21:10:22.689999 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.689977    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"]
Oct 30 21:10:22.692805 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.692786    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"]
Oct 30 21:10:22.754636 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754602    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.754840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754666    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.754840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754716    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.754931 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754858    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.754931 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754911    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755006 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754944    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndcn\" (UniqueName: \"kubernetes.io/projected/426cdbf2-0251-4f31-aea0-b393f86ce297-kube-api-access-5ndcn\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.755006 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.754981    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755108 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755018    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755164 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755146    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fx7b\" (UniqueName: \"kubernetes.io/projected/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-kube-api-access-8fx7b\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755253 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755233    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755326 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755245    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755326 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755268    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.755326 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755280    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755482 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755372    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755482 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755395    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.755482 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755450    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755657 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755505    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.755722 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755648    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.755813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755733    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.755882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755803    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755840    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.755882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755838    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.756042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.755894    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426cdbf2-0251-4f31-aea0-b393f86ce297-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.757461 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.757439    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.757567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.757532    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.812939 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.812871    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.813070 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.812947    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fx7b\" (UniqueName: \"kubernetes.io/projected/f2cc343a-deb9-49a0-8103-d3a4eeb8b892-kube-api-access-8fx7b\") pod \"maas-default-gateway-openshift-default-6886b96f9-f8t7j\" (UID: \"f2cc343a-deb9-49a0-8103-d3a4eeb8b892\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.857065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857030    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857069    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857270 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857242    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426cdbf2-0251-4f31-aea0-b393f86ce297-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857270 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857260    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857282    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857411    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857495 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857427    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcn\" (UniqueName: \"kubernetes.io/projected/426cdbf2-0251-4f31-aea0-b393f86ce297-kube-api-access-5ndcn\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857544 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857430    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857596 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857546    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857596 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857584    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857623    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.857940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857915    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426cdbf2-0251-4f31-aea0-b393f86ce297-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.858064 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857955    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.858064 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.857967    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.859669 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.859641    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.859669 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.859664    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.893440 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.893406    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:22.914278 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.914255    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426cdbf2-0251-4f31-aea0-b393f86ce297-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.914385 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.914365    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndcn\" (UniqueName: \"kubernetes.io/projected/426cdbf2-0251-4f31-aea0-b393f86ce297-kube-api-access-5ndcn\") pod \"maas-default-gateway-openshift-default-6886b96f9-d2bvn\" (UID: \"426cdbf2-0251-4f31-aea0-b393f86ce297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:22.945389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:22.945357    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:23.321565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.321530    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"]
Oct 30 21:10:23.359850 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:10:23.359814    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cc343a_deb9_49a0_8103_d3a4eeb8b892.slice/crio-9975f41e1888a0658921944992677ec75b04402c030bc7890b9deefe8140cb4d WatchSource:0}: Error finding container 9975f41e1888a0658921944992677ec75b04402c030bc7890b9deefe8140cb4d: Status 404 returned error can't find the container with id 9975f41e1888a0658921944992677ec75b04402c030bc7890b9deefe8140cb4d
Oct 30 21:10:23.361999 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.361967    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:23.362054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.362043    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:23.362097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.362073    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:23.374803 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.374776    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"]
Oct 30 21:10:23.377577 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:10:23.377552    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426cdbf2_0251_4f31_aea0_b393f86ce297.slice/crio-9d4c418fd8533d5aa38c7845bce0f627f58d39066ea7611f714063409e51b594 WatchSource:0}: Error finding container 9d4c418fd8533d5aa38c7845bce0f627f58d39066ea7611f714063409e51b594: Status 404 returned error can't find the container with id 9d4c418fd8533d5aa38c7845bce0f627f58d39066ea7611f714063409e51b594
Oct 30 21:10:23.379865 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.379834    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:23.379964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.379921    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:23.379964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.379953    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:23.715797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.715728    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" event={"ID":"f2cc343a-deb9-49a0-8103-d3a4eeb8b892","Type":"ContainerStarted","Data":"ced02cdea9d10dee477c0b193db2c9ca3eb83978cbd8a0315e832e7293650424"}
Oct 30 21:10:23.716310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.715851    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" event={"ID":"f2cc343a-deb9-49a0-8103-d3a4eeb8b892","Type":"ContainerStarted","Data":"9975f41e1888a0658921944992677ec75b04402c030bc7890b9deefe8140cb4d"}
Oct 30 21:10:23.717982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.717951    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" event={"ID":"426cdbf2-0251-4f31-aea0-b393f86ce297","Type":"ContainerStarted","Data":"fc2fc1435f0cadcbe2df3f5b7d077da236478a1bb5d2450cd9acec98612a4511"}
Oct 30 21:10:23.718111 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.718003    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" event={"ID":"426cdbf2-0251-4f31-aea0-b393f86ce297","Type":"ContainerStarted","Data":"9d4c418fd8533d5aa38c7845bce0f627f58d39066ea7611f714063409e51b594"}
Oct 30 21:10:23.898775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.897971    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:23.946193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.946151    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:23.958415 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.958362    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podStartSLOduration=1.958347624 podStartE2EDuration="1.958347624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:23.956676948 +0000 UTC m=+3004.557478910" watchObservedRunningTime="2026-10-30 21:10:23.958347624 +0000 UTC m=+3004.559149583"
Oct 30 21:10:23.959195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:23.959162    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podStartSLOduration=1.9591531070000001 podStartE2EDuration="1.959153107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:23.839231044 +0000 UTC m=+3004.440033005" watchObservedRunningTime="2026-10-30 21:10:23.959153107 +0000 UTC m=+3004.559955066"
Oct 30 21:10:24.896772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:24.895818    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 21:10:24.896772 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:24.895887    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 21:10:24.947853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:24.947817    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:10:24.948051 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:24.947884    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podUID="426cdbf2-0251-4f31-aea0-b393f86ce297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 21:10:25.894217 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:25.894177    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 21:10:25.895265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:25.895230    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 21:10:25.946647 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:25.946613    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:10:25.947064 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:25.946668    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podUID="426cdbf2-0251-4f31-aea0-b393f86ce297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 21:10:26.897261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:26.897221    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 21:10:26.897572 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:26.897538    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 21:10:26.946893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:26.946862    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:10:26.947392 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:26.947369    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podUID="426cdbf2-0251-4f31-aea0-b393f86ce297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 21:10:27.895199 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:27.895155    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 21:10:27.895528 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:27.895497    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 21:10:27.946934 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:27.946902    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:10:27.947405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:27.947380    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podUID="426cdbf2-0251-4f31-aea0-b393f86ce297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 21:10:28.894768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:28.894196    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 21:10:28.894768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:28.894262    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 21:10:28.946140 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:28.946101    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:10:28.946333 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:28.946158    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podUID="426cdbf2-0251-4f31-aea0-b393f86ce297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 21:10:29.802559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.802518    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"]
Oct 30 21:10:29.806610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.806585    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.864668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.864629    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"]
Oct 30 21:10:29.869362 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.869333    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.869538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.869386    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.869618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.869553    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmp6h\" (UniqueName: \"kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.894453 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.894405    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 21:10:29.894665 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.894479    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 21:10:29.911156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.911049    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:10:29.911156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.911099    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:10:29.911156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.911147    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:10:29.911717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.911695    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:10:29.946499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.946458    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:10:29.946679 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.946531    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn" podUID="426cdbf2-0251-4f31-aea0-b393f86ce297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 21:10:29.970260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.970230    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.970534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.970285    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.970534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.970355    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmp6h\" (UniqueName: \"kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.971166 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.971133    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:29.973789 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:29.973767    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:30.028690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:30.028649    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmp6h\" (UniqueName: \"kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h\") pod \"maas-api-revoke-sub-85237e45-vk9t2\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:30.122233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:30.122152    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:30.503390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:30.503251    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"]
Oct 30 21:10:30.507156 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:10:30.507123    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08638f3_fdd4_4632_950a_3295822f330c.slice/crio-5e2ab6aec9df0ec62478ed658b7fae50c76781277a55908839275bbe120ef9d6 WatchSource:0}: Error finding container 5e2ab6aec9df0ec62478ed658b7fae50c76781277a55908839275bbe120ef9d6: Status 404 returned error can't find the container with id 5e2ab6aec9df0ec62478ed658b7fae50c76781277a55908839275bbe120ef9d6
Oct 30 21:10:30.760604 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:30.760519    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2" event={"ID":"e08638f3-fdd4-4632-950a-3295822f330c","Type":"ContainerStarted","Data":"5e2ab6aec9df0ec62478ed658b7fae50c76781277a55908839275bbe120ef9d6"}
Oct 30 21:10:31.683800 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.683727    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:31.684322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.684190    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:31.685244 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.685215    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn"
Oct 30 21:10:31.716191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.716164    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:31.716642 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.716608    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:31.717419 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.717396    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j"
Oct 30 21:10:31.767352 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.767324    5967 generic.go:356] "Generic (PLEG): container finished" podID="e08638f3-fdd4-4632-950a-3295822f330c" containerID="c9e58d736ff06b2996e7ea9f837959c59214e1a10fd0edf0f786d1127ea7d301" exitCode=0
Oct 30 21:10:31.767526 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:31.767416    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2" event={"ID":"e08638f3-fdd4-4632-950a-3295822f330c","Type":"ContainerDied","Data":"c9e58d736ff06b2996e7ea9f837959c59214e1a10fd0edf0f786d1127ea7d301"}
Oct 30 21:10:31.987668 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:10:31.987632    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:10:32.915520 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:32.915496    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:33.004631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.004597    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca\") pod \"e08638f3-fdd4-4632-950a-3295822f330c\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") "
Oct 30 21:10:33.004863 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.004696    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca\") pod \"e08638f3-fdd4-4632-950a-3295822f330c\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") "
Oct 30 21:10:33.004863 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.004762    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h\" (UniqueName: \"kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h\") pod \"e08638f3-fdd4-4632-950a-3295822f330c\" (UID: \"e08638f3-fdd4-4632-950a-3295822f330c\") "
Oct 30 21:10:33.005148 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.005122    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca" pod "e08638f3-fdd4-4632-950a-3295822f330c" (UID: "e08638f3-fdd4-4632-950a-3295822f330c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:33.006951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.006922    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h" pod "e08638f3-fdd4-4632-950a-3295822f330c" (UID: "e08638f3-fdd4-4632-950a-3295822f330c"). InnerVolumeSpecName "kube-api-access-tmp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:33.007048 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.006959    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca" pod "e08638f3-fdd4-4632-950a-3295822f330c" (UID: "e08638f3-fdd4-4632-950a-3295822f330c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:33.105913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.105833    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:33.105913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.105862    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08638f3-fdd4-4632-950a-3295822f330c-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:33.105913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.105871    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmp6h\" (UniqueName: \"kubernetes.io/projected/e08638f3-fdd4-4632-950a-3295822f330c-kube-api-access-tmp6h\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:33.779485 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.779457    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"
Oct 30 21:10:33.779668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.779492    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2" event={"ID":"e08638f3-fdd4-4632-950a-3295822f330c","Type":"ContainerDied","Data":"5e2ab6aec9df0ec62478ed658b7fae50c76781277a55908839275bbe120ef9d6"}
Oct 30 21:10:33.779668 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:33.779544    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e2ab6aec9df0ec62478ed658b7fae50c76781277a55908839275bbe120ef9d6"
Oct 30 21:10:37.574657 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.574624    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"]
Oct 30 21:10:37.575127 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.575061    5967 cpu_manager.go:414] "containerMap: removing container" podUID="e08638f3-fdd4-4632-950a-3295822f330c" containerName="revoke-keys"
Oct 30 21:10:37.575127 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.575072    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08638f3-fdd4-4632-950a-3295822f330c" containerName="revoke-keys"
Oct 30 21:10:37.575215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.575189    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08638f3-fdd4-4632-950a-3295822f330c" containerName="revoke-keys"
Oct 30 21:10:37.578504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.578489    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.653979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.653941    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.653979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.653979    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654159    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xhr\" (UniqueName: \"kubernetes.io/projected/b0826915-086e-4d42-8ade-b517b6aa7925-kube-api-access-d6xhr\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654360 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654336    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0826915-086e-4d42-8ade-b517b6aa7925-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654368    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0826915-086e-4d42-8ade-b517b6aa7925-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654445    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0826915-086e-4d42-8ade-b517b6aa7925-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654472    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654545    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.654636 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.654589    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756089 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756045    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0826915-086e-4d42-8ade-b517b6aa7925-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756108    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0826915-086e-4d42-8ade-b517b6aa7925-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756135    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756161    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756186    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756223    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756249    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756324    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xhr\" (UniqueName: \"kubernetes.io/projected/b0826915-086e-4d42-8ade-b517b6aa7925-kube-api-access-d6xhr\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756441    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0826915-086e-4d42-8ade-b517b6aa7925-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756634    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756656    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756773    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.756944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.756922    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.757198 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.757176    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0826915-086e-4d42-8ade-b517b6aa7925-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.758693 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.758673    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0826915-086e-4d42-8ade-b517b6aa7925-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.758839 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.758677    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0826915-086e-4d42-8ade-b517b6aa7925-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.783037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.783007    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"]
Oct 30 21:10:37.812363 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.812326    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0826915-086e-4d42-8ade-b517b6aa7925-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.812566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.812544    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xhr\" (UniqueName: \"kubernetes.io/projected/b0826915-086e-4d42-8ade-b517b6aa7925-kube-api-access-d6xhr\") pod \"maas-default-gateway-openshift-default-6886b96f9-xbx2q\" (UID: \"b0826915-086e-4d42-8ade-b517b6aa7925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:37.891395 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:37.891308    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:38.319071 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.319034    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"]
Oct 30 21:10:38.357626 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:10:38.357598    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0826915_086e_4d42_8ade_b517b6aa7925.slice/crio-ab6723de04be79b7ace67c2c3f7041141b99dafa473b69302271448919423811 WatchSource:0}: Error finding container ab6723de04be79b7ace67c2c3f7041141b99dafa473b69302271448919423811: Status 404 returned error can't find the container with id ab6723de04be79b7ace67c2c3f7041141b99dafa473b69302271448919423811
Oct 30 21:10:38.359805 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.359771    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.359896 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.359837    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.359896 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.359865    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:38.820060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.820022    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" event={"ID":"b0826915-086e-4d42-8ade-b517b6aa7925","Type":"ContainerStarted","Data":"783c35aada0e1a123de8435529ebe27c66ee245fbc73933c8e66257d947e292a"}
Oct 30 21:10:38.820511 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.820079    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" event={"ID":"b0826915-086e-4d42-8ade-b517b6aa7925","Type":"ContainerStarted","Data":"ab6723de04be79b7ace67c2c3f7041141b99dafa473b69302271448919423811"}
Oct 30 21:10:38.892173 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.892076    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:38.988371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:38.988309    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podStartSLOduration=1.988292159 podStartE2EDuration="1.988292159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:38.987319818 +0000 UTC m=+3019.588121779" watchObservedRunningTime="2026-10-30 21:10:38.988292159 +0000 UTC m=+3019.589094119"
Oct 30 21:10:39.895625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:39.895590    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:10:39.896060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:39.895641    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:10:40.894137 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:40.894100    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:10:40.894390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:40.894155    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:10:41.892765 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:41.892221    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:10:41.892765 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:41.892300    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:10:42.538503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.538466    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-756d9799fc-cfpl2"]
Oct 30 21:10:42.539165 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.538715    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-756d9799fc-cfpl2" podUID="a4434f49-5004-40a0-9f58-b0371bbc5925" containerName="manager" containerID="cri-o://a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468" gracePeriod=30
Oct 30 21:10:42.819851 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.819827    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:42.842351 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.842317    5967 generic.go:356] "Generic (PLEG): container finished" podID="a4434f49-5004-40a0-9f58-b0371bbc5925" containerID="a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468" exitCode=0
Oct 30 21:10:42.842503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.842360    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-cfpl2"
Oct 30 21:10:42.842503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.842368    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-cfpl2" event={"ID":"a4434f49-5004-40a0-9f58-b0371bbc5925","Type":"ContainerDied","Data":"a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468"}
Oct 30 21:10:42.842503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.842410    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-cfpl2" event={"ID":"a4434f49-5004-40a0-9f58-b0371bbc5925","Type":"ContainerDied","Data":"60923f5cd85c1d4312fd85c33253e12646c1afc1f0f18bd36fcbe39a935d4180"}
Oct 30 21:10:42.842503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.842436    5967 scope.go:122] "RemoveContainer" containerID="a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468"
Oct 30 21:10:42.851524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.851506    5967 scope.go:122] "RemoveContainer" containerID="a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468"
Oct 30 21:10:42.851820 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:10:42.851802    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468\": container with ID starting with a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468 not found: ID does not exist" containerID="a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468"
Oct 30 21:10:42.851907 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.851833    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468"} err="failed to get container status \"a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468\": rpc error: code = NotFound desc = could not find container \"a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468\": container with ID starting with a3d93dc6dd1deef8c7f6879d20df187ce63dcb02af5a855b79e7e34ae1189468 not found: ID does not exist"
Oct 30 21:10:42.892032 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.891596    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:10:42.892032 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.891656    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:10:42.928570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.928533    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t\" (UniqueName: \"kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t\") pod \"a4434f49-5004-40a0-9f58-b0371bbc5925\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") "
Oct 30 21:10:42.929095 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.928678    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert\") pod \"a4434f49-5004-40a0-9f58-b0371bbc5925\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") "
Oct 30 21:10:42.929095 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.928756    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs\") pod \"a4434f49-5004-40a0-9f58-b0371bbc5925\" (UID: \"a4434f49-5004-40a0-9f58-b0371bbc5925\") "
Oct 30 21:10:42.930981 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.930959    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert" pod "a4434f49-5004-40a0-9f58-b0371bbc5925" (UID: "a4434f49-5004-40a0-9f58-b0371bbc5925"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:42.931386 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.931356    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs" pod "a4434f49-5004-40a0-9f58-b0371bbc5925" (UID: "a4434f49-5004-40a0-9f58-b0371bbc5925"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:42.931462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:42.931362    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t" pod "a4434f49-5004-40a0-9f58-b0371bbc5925" (UID: "a4434f49-5004-40a0-9f58-b0371bbc5925"). InnerVolumeSpecName "kube-api-access-lhp7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:43.030706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.030670    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp7t\" (UniqueName: \"kubernetes.io/projected/a4434f49-5004-40a0-9f58-b0371bbc5925-kube-api-access-lhp7t\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:43.030706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.030705    5967 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-webhook-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:43.030925 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.030717    5967 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4434f49-5004-40a0-9f58-b0371bbc5925-metrics-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:43.263699 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.263667    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-756d9799fc-cfpl2"]
Oct 30 21:10:43.267079 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.267054    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-756d9799fc-cfpl2"]
Oct 30 21:10:43.541664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.541630    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4434f49-5004-40a0-9f58-b0371bbc5925" path="/var/lib/kubelet/pods/a4434f49-5004-40a0-9f58-b0371bbc5925/volumes"
Oct 30 21:10:43.892243 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.892150    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:10:43.892534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:43.892505    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:10:44.707810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.707774    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-756d9799fc-b67rj"]
Oct 30 21:10:44.708248 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.708235    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a4434f49-5004-40a0-9f58-b0371bbc5925" containerName="manager"
Oct 30 21:10:44.708289 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.708249    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4434f49-5004-40a0-9f58-b0371bbc5925" containerName="manager"
Oct 30 21:10:44.708325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.708320    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4434f49-5004-40a0-9f58-b0371bbc5925" containerName="manager"
Oct 30 21:10:44.713003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.712979    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.771610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.771572    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d9799fc-b67rj"]
Oct 30 21:10:44.817304 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.817270    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:10:44.817304 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.817308    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:10:44.817549 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.817344    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:10:44.817549 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.817503    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-42mmb\""
Oct 30 21:10:44.849432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.849399    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9144ff77-360b-4134-b77d-e8e1f12005e9-webhook-cert\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.849621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.849443    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9144ff77-360b-4134-b77d-e8e1f12005e9-metrics-certs\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.849621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.849587    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcsk\" (UniqueName: \"kubernetes.io/projected/9144ff77-360b-4134-b77d-e8e1f12005e9-kube-api-access-7fcsk\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.892219 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.892182    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:10:44.892399 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.892234    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:10:44.950402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.950366    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9144ff77-360b-4134-b77d-e8e1f12005e9-webhook-cert\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.950402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.950407    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9144ff77-360b-4134-b77d-e8e1f12005e9-metrics-certs\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.950649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.950465    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcsk\" (UniqueName: \"kubernetes.io/projected/9144ff77-360b-4134-b77d-e8e1f12005e9-kube-api-access-7fcsk\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.953095 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.953068    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9144ff77-360b-4134-b77d-e8e1f12005e9-webhook-cert\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:44.953223 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:44.953111    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9144ff77-360b-4134-b77d-e8e1f12005e9-metrics-certs\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:45.006316 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:45.006280    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcsk\" (UniqueName: \"kubernetes.io/projected/9144ff77-360b-4134-b77d-e8e1f12005e9-kube-api-access-7fcsk\") pod \"maas-controller-756d9799fc-b67rj\" (UID: \"9144ff77-360b-4134-b77d-e8e1f12005e9\") " pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:45.026266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:45.026222    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:45.420492 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:45.420465    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d9799fc-b67rj"]
Oct 30 21:10:45.423009 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:10:45.422984    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9144ff77_360b_4134_b77d_e8e1f12005e9.slice/crio-a383437cc9c0eb41ccf8299137c9dfcf0446f0ba1f4afd6ad15a33fa7a8e4657 WatchSource:0}: Error finding container a383437cc9c0eb41ccf8299137c9dfcf0446f0ba1f4afd6ad15a33fa7a8e4657: Status 404 returned error can't find the container with id a383437cc9c0eb41ccf8299137c9dfcf0446f0ba1f4afd6ad15a33fa7a8e4657
Oct 30 21:10:45.863325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:45.863289    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-b67rj" event={"ID":"9144ff77-360b-4134-b77d-e8e1f12005e9","Type":"ContainerStarted","Data":"a383437cc9c0eb41ccf8299137c9dfcf0446f0ba1f4afd6ad15a33fa7a8e4657"}
Oct 30 21:10:46.360200 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:46.360168    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:46.360520 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:46.360495    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:46.361396 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:46.361380    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q"
Oct 30 21:10:46.870094 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:46.870053    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-b67rj" event={"ID":"9144ff77-360b-4134-b77d-e8e1f12005e9","Type":"ContainerStarted","Data":"dde7101c54bc30fab3df6a5c1ec673822e0069b050474ca52df2fbae17f36da9"}
Oct 30 21:10:46.870496 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:46.870145    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:10:46.987083 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:46.987025    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-756d9799fc-b67rj" podStartSLOduration=2.078737772 podStartE2EDuration="2.987010966s" totalImagesPullingTime="908.273194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:46.984940239 +0000 UTC m=+3027.585742201" watchObservedRunningTime="2026-10-30 21:10:46.987010966 +0000 UTC m=+3027.587812925"
Oct 30 21:10:57.879573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:10:57.879544    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-756d9799fc-b67rj"
Oct 30 21:11:01.979821 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:11:01.979773    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:11:13.046850 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:13.046803    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"]
Oct 30 21:11:13.095880 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:13.095839    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b0609d-c8nrs"]
Oct 30 21:11:13.097521 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:13.097492    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"]
Oct 30 21:11:13.098953 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:13.098933    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719cb10b-z94gq"]
Oct 30 21:11:13.541380 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:13.541346    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3005d396-c394-4ff8-8ecc-dd6d2274adda" path="/var/lib/kubelet/pods/3005d396-c394-4ff8-8ecc-dd6d2274adda/volumes"
Oct 30 21:11:13.541845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:13.541830    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbda826e-1568-4840-a766-6d6db04d3ca8" path="/var/lib/kubelet/pods/dbda826e-1568-4840-a766-6d6db04d3ca8/volumes"
Oct 30 21:11:22.657021 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:22.656976    5967 scope.go:122] "RemoveContainer" containerID="23aaca8080f37263a6a6d8eea478e0df44abd461d03d2be47c1d7a4127187d42"
Oct 30 21:11:22.666063 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:22.666044    5967 scope.go:122] "RemoveContainer" containerID="897bc8446388622e780602919b832b2ef9472f64ad7a1a0130a53f7ab82f37c4"
Oct 30 21:11:22.674116 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:22.674094    5967 scope.go:122] "RemoveContainer" containerID="788eebac4aa134a7fdd1a31aba0b4dc05b06c3192e78e0ddddffb1052e43beb5"
Oct 30 21:11:32.044468 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:11:32.044049    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:11:32.410827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:32.403865    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:32.410827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:32.403967    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q" podUID="b0826915-086e-4d42-8ade-b517b6aa7925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:32.762936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:32.762897    5967 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:32.763143 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:32.762971    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j" podUID="f2cc343a-deb9-49a0-8103-d3a4eeb8b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:38.489159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.489122    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"]
Oct 30 21:11:38.493944 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.493926    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.548106 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.548068    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:38.597566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.597517    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:38.597566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.597562    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:11:38.597839 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.597615    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:11:38.599487 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.599467    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"]
Oct 30 21:11:38.610646 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.610613    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.610861 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.610754    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.610861 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.610802    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsm2\" (UniqueName: \"kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.711755 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.711707    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.711946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.711791    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsm2\" (UniqueName: \"kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.711946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.711818    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.712354 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.712334    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.714443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.714417    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.768724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.768645    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsm2\" (UniqueName: \"kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2\") pod \"maas-api-revoke-sub-c1e271a9-xv524\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:38.806547 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:38.806512    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:39.178884 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:39.178856    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"]
Oct 30 21:11:39.181815 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:11:39.181785    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29408351_8883_4227_b164_9b0f06bf9e50.slice/crio-7b4aed2e680ef8b7778b1635899b13b4c4ce26cfd761f4e37ce007b80cdc375d WatchSource:0}: Error finding container 7b4aed2e680ef8b7778b1635899b13b4c4ce26cfd761f4e37ce007b80cdc375d: Status 404 returned error can't find the container with id 7b4aed2e680ef8b7778b1635899b13b4c4ce26cfd761f4e37ce007b80cdc375d
Oct 30 21:11:40.193842 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:40.193803    5967 generic.go:356] "Generic (PLEG): container finished" podID="29408351-8883-4227-b164-9b0f06bf9e50" containerID="e71523a4fac67cc729f9f12bb916fa02a28478b95e87e0b060548a0a67010d61" exitCode=0
Oct 30 21:11:40.194250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:40.193889    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524" event={"ID":"29408351-8883-4227-b164-9b0f06bf9e50","Type":"ContainerDied","Data":"e71523a4fac67cc729f9f12bb916fa02a28478b95e87e0b060548a0a67010d61"}
Oct 30 21:11:40.194250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:40.193942    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524" event={"ID":"29408351-8883-4227-b164-9b0f06bf9e50","Type":"ContainerStarted","Data":"7b4aed2e680ef8b7778b1635899b13b4c4ce26cfd761f4e37ce007b80cdc375d"}
Oct 30 21:11:41.341963 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.341936    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:41.439096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.439067    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca\") pod \"29408351-8883-4227-b164-9b0f06bf9e50\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") "
Oct 30 21:11:41.439286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.439123    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2\" (UniqueName: \"kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2\") pod \"29408351-8883-4227-b164-9b0f06bf9e50\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") "
Oct 30 21:11:41.439286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.439164    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca\") pod \"29408351-8883-4227-b164-9b0f06bf9e50\" (UID: \"29408351-8883-4227-b164-9b0f06bf9e50\") "
Oct 30 21:11:41.439577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.439551    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca" pod "29408351-8883-4227-b164-9b0f06bf9e50" (UID: "29408351-8883-4227-b164-9b0f06bf9e50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:41.439728 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.439659    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:41.441280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.441261    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca" pod "29408351-8883-4227-b164-9b0f06bf9e50" (UID: "29408351-8883-4227-b164-9b0f06bf9e50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:41.441532 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.441516    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2" pod "29408351-8883-4227-b164-9b0f06bf9e50" (UID: "29408351-8883-4227-b164-9b0f06bf9e50"). InnerVolumeSpecName "kube-api-access-bmsm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:41.540198 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.540166    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29408351-8883-4227-b164-9b0f06bf9e50-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:41.540198 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:41.540193    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmsm2\" (UniqueName: \"kubernetes.io/projected/29408351-8883-4227-b164-9b0f06bf9e50-kube-api-access-bmsm2\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.206147 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:42.206112    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524" event={"ID":"29408351-8883-4227-b164-9b0f06bf9e50","Type":"ContainerDied","Data":"7b4aed2e680ef8b7778b1635899b13b4c4ce26cfd761f4e37ce007b80cdc375d"}
Oct 30 21:11:42.206147 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:42.206141    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"
Oct 30 21:11:42.206384 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:11:42.206164    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b4aed2e680ef8b7778b1635899b13b4c4ce26cfd761f4e37ce007b80cdc375d"
Oct 30 21:12:01.981076 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:12:01.981037    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:12:10.085665 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.085621    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"]
Oct 30 21:12:10.086218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.086157    5967 cpu_manager.go:414] "containerMap: removing container" podUID="29408351-8883-4227-b164-9b0f06bf9e50" containerName="revoke-keys"
Oct 30 21:12:10.086218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.086173    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29408351-8883-4227-b164-9b0f06bf9e50" containerName="revoke-keys"
Oct 30 21:12:10.086316 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.086302    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="29408351-8883-4227-b164-9b0f06bf9e50" containerName="revoke-keys"
Oct 30 21:12:10.089853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.089826    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.145300 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.145270    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"]
Oct 30 21:12:10.165945 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.165905    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvqq\" (UniqueName: \"kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.166135 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.166048    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.166180 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.166135    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.192047 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.192007    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:10.192218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.192053    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:12:10.192218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.192052    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:12:10.192218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.192109    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:12:10.267038 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.266998    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.267268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.267065    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.267268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.267138    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvqq\" (UniqueName: \"kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.267833 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.267807    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.269768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.269722    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.324275 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.324240    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvqq\" (UniqueName: \"kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq\") pod \"maas-api-revoke-sub-2e1f5b5f-xhp4v\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.404297 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.404206    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:10.778485 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:10.778356    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"]
Oct 30 21:12:10.781905 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:12:10.781871    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d655e45_b9c0_4368_ab89_1e21cfc739b4.slice/crio-fa3dda8467004251a193182454c61516cb4089ef70a91563d809538e2c9f5a37 WatchSource:0}: Error finding container fa3dda8467004251a193182454c61516cb4089ef70a91563d809538e2c9f5a37: Status 404 returned error can't find the container with id fa3dda8467004251a193182454c61516cb4089ef70a91563d809538e2c9f5a37
Oct 30 21:12:11.387148 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:11.387060    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v" event={"ID":"2d655e45-b9c0-4368-ab89-1e21cfc739b4","Type":"ContainerStarted","Data":"8dc9d8458cddc0e24d6ac4b5896f77e45b580b3459e40010e10f66953df9ad44"}
Oct 30 21:12:11.387148 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:11.387112    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v" event={"ID":"2d655e45-b9c0-4368-ab89-1e21cfc739b4","Type":"ContainerStarted","Data":"fa3dda8467004251a193182454c61516cb4089ef70a91563d809538e2c9f5a37"}
Oct 30 21:12:11.501554 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:11.501491    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v" podStartSLOduration=1.5014747800000001 podStartE2EDuration="1.50147478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:11.500049253 +0000 UTC m=+3112.100851208" watchObservedRunningTime="2026-10-30 21:12:11.50147478 +0000 UTC m=+3112.102276741"
Oct 30 21:12:12.393672 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:12.393638    5967 generic.go:356] "Generic (PLEG): container finished" podID="2d655e45-b9c0-4368-ab89-1e21cfc739b4" containerID="8dc9d8458cddc0e24d6ac4b5896f77e45b580b3459e40010e10f66953df9ad44" exitCode=0
Oct 30 21:12:12.394144 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:12.393727    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v" event={"ID":"2d655e45-b9c0-4368-ab89-1e21cfc739b4","Type":"ContainerDied","Data":"8dc9d8458cddc0e24d6ac4b5896f77e45b580b3459e40010e10f66953df9ad44"}
Oct 30 21:12:13.547885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.547853    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:13.603216 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.603179    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca\") pod \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") "
Oct 30 21:12:13.603216 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.603222    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca\") pod \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") "
Oct 30 21:12:13.603434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.603282    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq\" (UniqueName: \"kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq\") pod \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\" (UID: \"2d655e45-b9c0-4368-ab89-1e21cfc739b4\") "
Oct 30 21:12:13.603690 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.603665    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca" pod "2d655e45-b9c0-4368-ab89-1e21cfc739b4" (UID: "2d655e45-b9c0-4368-ab89-1e21cfc739b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:13.605618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.605591    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca" pod "2d655e45-b9c0-4368-ab89-1e21cfc739b4" (UID: "2d655e45-b9c0-4368-ab89-1e21cfc739b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:13.605704 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.605670    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq" pod "2d655e45-b9c0-4368-ab89-1e21cfc739b4" (UID: "2d655e45-b9c0-4368-ab89-1e21cfc739b4"). InnerVolumeSpecName "kube-api-access-7pvqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:13.705258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.705165    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:13.705258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.705200    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d655e45-b9c0-4368-ab89-1e21cfc739b4-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:13.705258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:13.705212    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvqq\" (UniqueName: \"kubernetes.io/projected/2d655e45-b9c0-4368-ab89-1e21cfc739b4-kube-api-access-7pvqq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:14.409591 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:14.409564    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"
Oct 30 21:12:14.409821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:14.409551    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v" event={"ID":"2d655e45-b9c0-4368-ab89-1e21cfc739b4","Type":"ContainerDied","Data":"fa3dda8467004251a193182454c61516cb4089ef70a91563d809538e2c9f5a37"}
Oct 30 21:12:14.409821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:14.409726    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa3dda8467004251a193182454c61516cb4089ef70a91563d809538e2c9f5a37"
Oct 30 21:12:22.534977 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.534926    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"]
Oct 30 21:12:22.535436 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.535424    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2d655e45-b9c0-4368-ab89-1e21cfc739b4" containerName="revoke-keys"
Oct 30 21:12:22.535490 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.535438    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d655e45-b9c0-4368-ab89-1e21cfc739b4" containerName="revoke-keys"
Oct 30 21:12:22.535546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.535536    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d655e45-b9c0-4368-ab89-1e21cfc739b4" containerName="revoke-keys"
Oct 30 21:12:22.538988 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.538963    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.594240 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.594209    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:22.594240 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.594244    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:12:22.643675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.643633    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:12:22.643876 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.643758    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:12:22.645903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.645881    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"]
Oct 30 21:12:22.704468 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.704427    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjfl\" (UniqueName: \"kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.704699 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.704530    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.704821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.704717    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.706426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.706400    5967 scope.go:122] "RemoveContainer" containerID="355c7f1fb5f1fcedb7c89f6a4da19e08b577fa882e0f94b653c8291c270a59d9"
Oct 30 21:12:22.716214 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.716193    5967 scope.go:122] "RemoveContainer" containerID="1ad59ee2addb518abae8a3f39aca1d139e09f08451f03bd05e9d159443bdf59e"
Oct 30 21:12:22.725820 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.725794    5967 scope.go:122] "RemoveContainer" containerID="d852df9c861d19660d75fb6b69659ba1e4c690a8e345a33f3603c72c178b993d"
Oct 30 21:12:22.806408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.806320    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.806579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.806426    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjfl\" (UniqueName: \"kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.806579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.806503    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.807218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.807193    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.808952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.808927    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:22.863534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:22.863499    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjfl\" (UniqueName: \"kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl\") pod \"maas-api-revoke-sub-ec843018-f5zmr\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:23.152896 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:23.152811    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:23.528695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:23.528528    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"]
Oct 30 21:12:23.531433 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:12:23.531401    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6aaaed_57e1_4cf2_ada8_f510194c4359.slice/crio-de33b363fa833d4c7d88f76941014feff76a8a3a6f7c14d7525e7621a8124340 WatchSource:0}: Error finding container de33b363fa833d4c7d88f76941014feff76a8a3a6f7c14d7525e7621a8124340: Status 404 returned error can't find the container with id de33b363fa833d4c7d88f76941014feff76a8a3a6f7c14d7525e7621a8124340
Oct 30 21:12:24.466810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:24.466772    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr" event={"ID":"3b6aaaed-57e1-4cf2-ada8-f510194c4359","Type":"ContainerStarted","Data":"238730b5d7cceb9b31b301df1db32330a867e9eab5a144c4e32bffc291db3f7d"}
Oct 30 21:12:24.467317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:24.466827    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr" event={"ID":"3b6aaaed-57e1-4cf2-ada8-f510194c4359","Type":"ContainerStarted","Data":"de33b363fa833d4c7d88f76941014feff76a8a3a6f7c14d7525e7621a8124340"}
Oct 30 21:12:24.581766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:24.581698    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr" podStartSLOduration=2.581681315 podStartE2EDuration="2.581681315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:24.580135107 +0000 UTC m=+3125.180937068" watchObservedRunningTime="2026-10-30 21:12:24.581681315 +0000 UTC m=+3125.182483278"
Oct 30 21:12:25.473757 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:25.473715    5967 generic.go:356] "Generic (PLEG): container finished" podID="3b6aaaed-57e1-4cf2-ada8-f510194c4359" containerID="238730b5d7cceb9b31b301df1db32330a867e9eab5a144c4e32bffc291db3f7d" exitCode=0
Oct 30 21:12:25.474276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:25.473810    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr" event={"ID":"3b6aaaed-57e1-4cf2-ada8-f510194c4359","Type":"ContainerDied","Data":"238730b5d7cceb9b31b301df1db32330a867e9eab5a144c4e32bffc291db3f7d"}
Oct 30 21:12:26.629393 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.629368    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:26.747004 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.746903    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca\") pod \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") "
Oct 30 21:12:26.747181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.747008    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca\") pod \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") "
Oct 30 21:12:26.747181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.747044    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl\" (UniqueName: \"kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl\") pod \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\" (UID: \"3b6aaaed-57e1-4cf2-ada8-f510194c4359\") "
Oct 30 21:12:26.747486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.747455    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca" pod "3b6aaaed-57e1-4cf2-ada8-f510194c4359" (UID: "3b6aaaed-57e1-4cf2-ada8-f510194c4359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:26.749464 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.749434    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca" pod "3b6aaaed-57e1-4cf2-ada8-f510194c4359" (UID: "3b6aaaed-57e1-4cf2-ada8-f510194c4359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:26.749641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.749619    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl" pod "3b6aaaed-57e1-4cf2-ada8-f510194c4359" (UID: "3b6aaaed-57e1-4cf2-ada8-f510194c4359"). InnerVolumeSpecName "kube-api-access-lnjfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:26.848708 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.848650    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:26.848708 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.848709    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6aaaed-57e1-4cf2-ada8-f510194c4359-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:26.848708 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:26.848730    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnjfl\" (UniqueName: \"kubernetes.io/projected/3b6aaaed-57e1-4cf2-ada8-f510194c4359-kube-api-access-lnjfl\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.488641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:27.488604    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr" event={"ID":"3b6aaaed-57e1-4cf2-ada8-f510194c4359","Type":"ContainerDied","Data":"de33b363fa833d4c7d88f76941014feff76a8a3a6f7c14d7525e7621a8124340"}
Oct 30 21:12:27.488865 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:27.488652    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"
Oct 30 21:12:27.488865 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:27.488663    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de33b363fa833d4c7d88f76941014feff76a8a3a6f7c14d7525e7621a8124340"
Oct 30 21:12:31.082234 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:31.082196    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"]
Oct 30 21:12:31.089475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:31.089441    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7237cd86-cdrrc"]
Oct 30 21:12:31.542212 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:31.542171    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92482e4b-ea84-4c9d-832c-23d893cde9c5" path="/var/lib/kubelet/pods/92482e4b-ea84-4c9d-832c-23d893cde9c5/volumes"
Oct 30 21:12:31.982755 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:12:31.982702    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:12:44.054788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:44.053421    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"]
Oct 30 21:12:44.105773 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:44.105237    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d3b76c-m9hrl"]
Oct 30 21:12:45.541831 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:12:45.541791    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983037f4-4245-4ee9-bbc4-64e7f0a01cb1" path="/var/lib/kubelet/pods/983037f4-4245-4ee9-bbc4-64e7f0a01cb1/volumes"
Oct 30 21:13:01.155488 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.155410    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"]
Oct 30 21:13:01.155952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.155938    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3b6aaaed-57e1-4cf2-ada8-f510194c4359" containerName="revoke-keys"
Oct 30 21:13:01.156003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.155952    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6aaaed-57e1-4cf2-ada8-f510194c4359" containerName="revoke-keys"
Oct 30 21:13:01.156039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.156027    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6aaaed-57e1-4cf2-ada8-f510194c4359" containerName="revoke-keys"
Oct 30 21:13:01.159390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.159374    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.211027 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.210995    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:13:01.227666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.227627    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.227895 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.227870    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.227960 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.227927    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2f5q\" (UniqueName: \"kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.260856 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.260818    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:01.261034 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.260928    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:01.262265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.262250    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:01.262801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.262779    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"]
Oct 30 21:13:01.328799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.328722    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.329012 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.328814    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2f5q\" (UniqueName: \"kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.329012 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.328888    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.329499 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.329432    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.331189 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.331167    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.385477 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.385439    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2f5q\" (UniqueName: \"kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q\") pod \"maas-api-revoke-sub-046e1343-smx8z\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.472696 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.472662    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:01.844154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:01.844111    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"]
Oct 30 21:13:01.846817 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:13:01.846789    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f232a_e86b_4089_b454_6060d6892dc3.slice/crio-1f75899541c924ca8403eafbadf2e5b44f6855d7b2ff98abd4ad71f458ffc456 WatchSource:0}: Error finding container 1f75899541c924ca8403eafbadf2e5b44f6855d7b2ff98abd4ad71f458ffc456: Status 404 returned error can't find the container with id 1f75899541c924ca8403eafbadf2e5b44f6855d7b2ff98abd4ad71f458ffc456
Oct 30 21:13:01.980260 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:13:01.980225    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:13:02.712770 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:02.712699    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z" event={"ID":"405f232a-e86b-4089-b454-6060d6892dc3","Type":"ContainerStarted","Data":"88846bd3d9d9a2c2ddece1785c53dfa3f497515327be13ce6ae5ac8f077ba184"}
Oct 30 21:13:02.713206 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:02.712810    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z" event={"ID":"405f232a-e86b-4089-b454-6060d6892dc3","Type":"ContainerStarted","Data":"1f75899541c924ca8403eafbadf2e5b44f6855d7b2ff98abd4ad71f458ffc456"}
Oct 30 21:13:02.830178 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:02.830127    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z" podStartSLOduration=1.8301133 podStartE2EDuration="1.8301133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:02.827445719 +0000 UTC m=+3163.428247678" watchObservedRunningTime="2026-10-30 21:13:02.8301133 +0000 UTC m=+3163.430915261"
Oct 30 21:13:03.719257 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:03.719223    5967 generic.go:356] "Generic (PLEG): container finished" podID="405f232a-e86b-4089-b454-6060d6892dc3" containerID="88846bd3d9d9a2c2ddece1785c53dfa3f497515327be13ce6ae5ac8f077ba184" exitCode=0
Oct 30 21:13:03.719686 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:03.719266    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z" event={"ID":"405f232a-e86b-4089-b454-6060d6892dc3","Type":"ContainerDied","Data":"88846bd3d9d9a2c2ddece1785c53dfa3f497515327be13ce6ae5ac8f077ba184"}
Oct 30 21:13:04.869536 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.869511    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:04.970654 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.970619    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q\" (UniqueName: \"kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q\") pod \"405f232a-e86b-4089-b454-6060d6892dc3\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") "
Oct 30 21:13:04.970866 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.970669    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca\") pod \"405f232a-e86b-4089-b454-6060d6892dc3\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") "
Oct 30 21:13:04.970866 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.970706    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca\") pod \"405f232a-e86b-4089-b454-6060d6892dc3\" (UID: \"405f232a-e86b-4089-b454-6060d6892dc3\") "
Oct 30 21:13:04.971111 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.971086    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca" pod "405f232a-e86b-4089-b454-6060d6892dc3" (UID: "405f232a-e86b-4089-b454-6060d6892dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:04.971234 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.971177    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:04.972937 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.972906    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q" pod "405f232a-e86b-4089-b454-6060d6892dc3" (UID: "405f232a-e86b-4089-b454-6060d6892dc3"). InnerVolumeSpecName "kube-api-access-k2f5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:04.973028 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:04.972969    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca" pod "405f232a-e86b-4089-b454-6060d6892dc3" (UID: "405f232a-e86b-4089-b454-6060d6892dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:05.072265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:05.072232    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2f5q\" (UniqueName: \"kubernetes.io/projected/405f232a-e86b-4089-b454-6060d6892dc3-kube-api-access-k2f5q\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:05.072265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:05.072261    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405f232a-e86b-4089-b454-6060d6892dc3-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:05.731880 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:05.731845    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"
Oct 30 21:13:05.732057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:05.731876    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z" event={"ID":"405f232a-e86b-4089-b454-6060d6892dc3","Type":"ContainerDied","Data":"1f75899541c924ca8403eafbadf2e5b44f6855d7b2ff98abd4ad71f458ffc456"}
Oct 30 21:13:05.732057 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:05.731947    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f75899541c924ca8403eafbadf2e5b44f6855d7b2ff98abd4ad71f458ffc456"
Oct 30 21:13:11.915481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:11.915445    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"]
Oct 30 21:13:11.915999 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:11.915982    5967 cpu_manager.go:414] "containerMap: removing container" podUID="405f232a-e86b-4089-b454-6060d6892dc3" containerName="revoke-keys"
Oct 30 21:13:11.916068 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:11.916003    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405f232a-e86b-4089-b454-6060d6892dc3" containerName="revoke-keys"
Oct 30 21:13:11.916107 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:11.916070    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="405f232a-e86b-4089-b454-6060d6892dc3" containerName="revoke-keys"
Oct 30 21:13:11.919358 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:11.919339    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:11.971662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:11.971630    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:12.021083 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.021053    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:12.021321 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.021121    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:12.021321 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.021226    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:13:12.023233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.023215    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"]
Oct 30 21:13:12.042866 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.042831    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.042994 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.042946    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2st\" (UniqueName: \"kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.043054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.043027    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.144322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.144287    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2st\" (UniqueName: \"kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.144518 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.144369    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.144518 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.144418    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.145110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.145080    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.146961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.146936    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.201659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.201579    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2st\" (UniqueName: \"kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st\") pod \"maas-api-revoke-sub-d3fba056-79tz7\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.232563 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.232530    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:12.610260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.610226    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"]
Oct 30 21:13:12.612069 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:13:12.612042    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6f59b4_c6da_411a_a4c4_b5821e03ceb8.slice/crio-4738889c5c3b1793bc20894b1786ecd7ec6f3469337e705f95c816977173e2b2 WatchSource:0}: Error finding container 4738889c5c3b1793bc20894b1786ecd7ec6f3469337e705f95c816977173e2b2: Status 404 returned error can't find the container with id 4738889c5c3b1793bc20894b1786ecd7ec6f3469337e705f95c816977173e2b2
Oct 30 21:13:12.778249 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:12.778216    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7" event={"ID":"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8","Type":"ContainerStarted","Data":"4738889c5c3b1793bc20894b1786ecd7ec6f3469337e705f95c816977173e2b2"}
Oct 30 21:13:13.785473 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:13.785435    5967 generic.go:356] "Generic (PLEG): container finished" podID="eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" containerID="9bbec052d4589a3dc272859aedadea321f0ab98c75b5c780abbaf6a7bd335d55" exitCode=0
Oct 30 21:13:13.786004 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:13.785526    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7" event={"ID":"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8","Type":"ContainerDied","Data":"9bbec052d4589a3dc272859aedadea321f0ab98c75b5c780abbaf6a7bd335d55"}
Oct 30 21:13:14.957229 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:14.957196    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:15.078815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.076811    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca\") pod \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") "
Oct 30 21:13:15.078815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.076878    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st\" (UniqueName: \"kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st\") pod \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") "
Oct 30 21:13:15.078815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.077069    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca\") pod \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\" (UID: \"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8\") "
Oct 30 21:13:15.078815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.077870    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca" pod "eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" (UID: "eb6f59b4-c6da-411a-a4c4-b5821e03ceb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:15.101213 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.092176    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st" pod "eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" (UID: "eb6f59b4-c6da-411a-a4c4-b5821e03ceb8"). InnerVolumeSpecName "kube-api-access-tm2st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:15.115780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.102870    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca" pod "eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" (UID: "eb6f59b4-c6da-411a-a4c4-b5821e03ceb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:15.180590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.180502    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:15.180590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.180543    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm2st\" (UniqueName: \"kubernetes.io/projected/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-kube-api-access-tm2st\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:15.180590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.180559    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:15.850327 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.850228    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7" event={"ID":"eb6f59b4-c6da-411a-a4c4-b5821e03ceb8","Type":"ContainerDied","Data":"4738889c5c3b1793bc20894b1786ecd7ec6f3469337e705f95c816977173e2b2"}
Oct 30 21:13:15.850588 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.850574    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4738889c5c3b1793bc20894b1786ecd7ec6f3469337e705f95c816977173e2b2"
Oct 30 21:13:15.850756 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:15.850725    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"
Oct 30 21:13:31.991425 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:13:31.991378    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:13:49.940972 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:49.940930    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"]
Oct 30 21:13:49.941633 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:49.941614    5967 cpu_manager.go:414] "containerMap: removing container" podUID="eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" containerName="revoke-keys"
Oct 30 21:13:49.941633 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:49.941634    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" containerName="revoke-keys"
Oct 30 21:13:49.941863 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:49.941765    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" containerName="revoke-keys"
Oct 30 21:13:49.945447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:49.945428    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.003352 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.003309    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"]
Oct 30 21:13:50.049096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.049048    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:50.049096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.049057    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:50.049096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.049091    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:50.049359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.049178    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:13:50.123828 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.123789    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q56b\" (UniqueName: \"kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.124037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.123920    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.124037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.123985    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.225110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.225073    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.225317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.225145    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q56b\" (UniqueName: \"kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.225317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.225208    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.225814 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.225789    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.227724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.227700    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.282562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.282531    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q56b\" (UniqueName: \"kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b\") pod \"maas-api-revoke-sub-ae594979-lnmvd\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.558957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.558880    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:50.933280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:50.933254    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"]
Oct 30 21:13:50.935385 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:13:50.935357    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48cd39b_41fc_47c5_8fbd_16fd1008c261.slice/crio-c7bfb1a766ed7f899f2d129875b27c4dd4264aeca1f372ce329b376a794900fc WatchSource:0}: Error finding container c7bfb1a766ed7f899f2d129875b27c4dd4264aeca1f372ce329b376a794900fc: Status 404 returned error can't find the container with id c7bfb1a766ed7f899f2d129875b27c4dd4264aeca1f372ce329b376a794900fc
Oct 30 21:13:51.054364 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:51.054328    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd" event={"ID":"c48cd39b-41fc-47c5-8fbd-16fd1008c261","Type":"ContainerStarted","Data":"c7bfb1a766ed7f899f2d129875b27c4dd4264aeca1f372ce329b376a794900fc"}
Oct 30 21:13:52.060857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:52.060819    5967 generic.go:356] "Generic (PLEG): container finished" podID="c48cd39b-41fc-47c5-8fbd-16fd1008c261" containerID="8ba356351b94155a08d2b7eac0182be43f5dc8abbddc8e6b8f4793342cda961c" exitCode=0
Oct 30 21:13:52.061290 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:52.060881    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd" event={"ID":"c48cd39b-41fc-47c5-8fbd-16fd1008c261","Type":"ContainerDied","Data":"8ba356351b94155a08d2b7eac0182be43f5dc8abbddc8e6b8f4793342cda961c"}
Oct 30 21:13:53.215193 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.215166    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:53.254112 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.254064    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca\") pod \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") "
Oct 30 21:13:53.254308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.254170    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca\") pod \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") "
Oct 30 21:13:53.254308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.254288    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b\" (UniqueName: \"kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b\") pod \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\" (UID: \"c48cd39b-41fc-47c5-8fbd-16fd1008c261\") "
Oct 30 21:13:53.254545 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.254520    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca" pod "c48cd39b-41fc-47c5-8fbd-16fd1008c261" (UID: "c48cd39b-41fc-47c5-8fbd-16fd1008c261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:53.254662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.254634    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:53.256442 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.256413    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b" pod "c48cd39b-41fc-47c5-8fbd-16fd1008c261" (UID: "c48cd39b-41fc-47c5-8fbd-16fd1008c261"). InnerVolumeSpecName "kube-api-access-4q56b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:53.256557 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.256486    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca" pod "c48cd39b-41fc-47c5-8fbd-16fd1008c261" (UID: "c48cd39b-41fc-47c5-8fbd-16fd1008c261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:53.355467 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.355386    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q56b\" (UniqueName: \"kubernetes.io/projected/c48cd39b-41fc-47c5-8fbd-16fd1008c261-kube-api-access-4q56b\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:53.355467 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:53.355419    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48cd39b-41fc-47c5-8fbd-16fd1008c261-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:54.073755 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:54.073696    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"
Oct 30 21:13:54.073982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:54.073692    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd" event={"ID":"c48cd39b-41fc-47c5-8fbd-16fd1008c261","Type":"ContainerDied","Data":"c7bfb1a766ed7f899f2d129875b27c4dd4264aeca1f372ce329b376a794900fc"}
Oct 30 21:13:54.073982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:13:54.073843    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7bfb1a766ed7f899f2d129875b27c4dd4264aeca1f372ce329b376a794900fc"
Oct 30 21:14:01.980758 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:14:01.980694    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:14:18.100594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.100558    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"]
Oct 30 21:14:18.101085 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.101070    5967 cpu_manager.go:414] "containerMap: removing container" podUID="c48cd39b-41fc-47c5-8fbd-16fd1008c261" containerName="revoke-keys"
Oct 30 21:14:18.101167 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.101085    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48cd39b-41fc-47c5-8fbd-16fd1008c261" containerName="revoke-keys"
Oct 30 21:14:18.101167 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.101163    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48cd39b-41fc-47c5-8fbd-16fd1008c261" containerName="revoke-keys"
Oct 30 21:14:18.104806 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.104768    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.208345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.208304    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:18.208345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.208321    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:18.208570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.208349    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:18.208570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.208433    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:14:18.210662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.210632    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"]
Oct 30 21:14:18.232539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.232510    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.232841 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.232617    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.232841 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.232716    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfq9\" (UniqueName: \"kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.334191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.334156    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.334395 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.334218    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.334395 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.334286    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfq9\" (UniqueName: \"kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.334904 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.334880    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.336731 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.336710    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.392025 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.391947    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfq9\" (UniqueName: \"kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9\") pod \"maas-api-revoke-sub-a4aef7d7-q82r5\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.418641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.418596    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:18.795094 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:18.795067    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"]
Oct 30 21:14:18.797548 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:14:18.797519    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfca4389_3d1a_46a5_9d17_7a540962de64.slice/crio-f77686cda1fcb670d2f61271a0ded597135a9525ec2d36d9d18d655e460da416 WatchSource:0}: Error finding container f77686cda1fcb670d2f61271a0ded597135a9525ec2d36d9d18d655e460da416: Status 404 returned error can't find the container with id f77686cda1fcb670d2f61271a0ded597135a9525ec2d36d9d18d655e460da416
Oct 30 21:14:19.223069 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:19.223030    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5" event={"ID":"cfca4389-3d1a-46a5-9d17-7a540962de64","Type":"ContainerStarted","Data":"38651d74d975f7609e6fc290056bf9376e5368a46cfafa874dc9aea2110ae1f8"}
Oct 30 21:14:19.223595 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:19.223094    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5" event={"ID":"cfca4389-3d1a-46a5-9d17-7a540962de64","Type":"ContainerStarted","Data":"f77686cda1fcb670d2f61271a0ded597135a9525ec2d36d9d18d655e460da416"}
Oct 30 21:14:19.337658 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:19.337543    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5" podStartSLOduration=1.33752389 podStartE2EDuration="1.33752389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:19.336430037 +0000 UTC m=+3239.937231997" watchObservedRunningTime="2026-10-30 21:14:19.33752389 +0000 UTC m=+3239.938325852"
Oct 30 21:14:20.230397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:20.230359    5967 generic.go:356] "Generic (PLEG): container finished" podID="cfca4389-3d1a-46a5-9d17-7a540962de64" containerID="38651d74d975f7609e6fc290056bf9376e5368a46cfafa874dc9aea2110ae1f8" exitCode=0
Oct 30 21:14:20.230397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:20.230404    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5" event={"ID":"cfca4389-3d1a-46a5-9d17-7a540962de64","Type":"ContainerDied","Data":"38651d74d975f7609e6fc290056bf9376e5368a46cfafa874dc9aea2110ae1f8"}
Oct 30 21:14:21.377967 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.377933    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:21.469204 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.469170    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca\") pod \"cfca4389-3d1a-46a5-9d17-7a540962de64\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") "
Oct 30 21:14:21.469416 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.469271    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9\" (UniqueName: \"kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9\") pod \"cfca4389-3d1a-46a5-9d17-7a540962de64\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") "
Oct 30 21:14:21.469416 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.469411    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca\") pod \"cfca4389-3d1a-46a5-9d17-7a540962de64\" (UID: \"cfca4389-3d1a-46a5-9d17-7a540962de64\") "
Oct 30 21:14:21.469726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.469691    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca" pod "cfca4389-3d1a-46a5-9d17-7a540962de64" (UID: "cfca4389-3d1a-46a5-9d17-7a540962de64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:21.469893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.469808    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:21.471538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.471517    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9" pod "cfca4389-3d1a-46a5-9d17-7a540962de64" (UID: "cfca4389-3d1a-46a5-9d17-7a540962de64"). InnerVolumeSpecName "kube-api-access-wsfq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:21.471607 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.471581    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca" pod "cfca4389-3d1a-46a5-9d17-7a540962de64" (UID: "cfca4389-3d1a-46a5-9d17-7a540962de64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:21.571247 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.571210    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca4389-3d1a-46a5-9d17-7a540962de64-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:21.571247 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:21.571242    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsfq9\" (UniqueName: \"kubernetes.io/projected/cfca4389-3d1a-46a5-9d17-7a540962de64-kube-api-access-wsfq9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:22.242819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:22.242773    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5" event={"ID":"cfca4389-3d1a-46a5-9d17-7a540962de64","Type":"ContainerDied","Data":"f77686cda1fcb670d2f61271a0ded597135a9525ec2d36d9d18d655e460da416"}
Oct 30 21:14:22.244941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:22.242836    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f77686cda1fcb670d2f61271a0ded597135a9525ec2d36d9d18d655e460da416"
Oct 30 21:14:22.244941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:22.242793    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"
Oct 30 21:14:22.767285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:22.767252    5967 scope.go:122] "RemoveContainer" containerID="2ad3a44c5cff6b75ce4d2e184dbadab263f6f01ef1dca52b67400089c52ea799"
Oct 30 21:14:22.776600 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:22.776579    5967 scope.go:122] "RemoveContainer" containerID="34e4e84e8fbec6436da94ac43d30afb6bca5110867f48f65120851eb3f02705c"
Oct 30 21:14:22.785572 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:22.785551    5967 scope.go:122] "RemoveContainer" containerID="69a91d4aad9ea3d5560085e2029fca96476530ea619a35f51338611e9cbb5e8e"
Oct 30 21:14:31.981489 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:14:31.981402    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:14:37.046593 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:37.046560    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"]
Oct 30 21:14:37.096429 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:37.096391    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826541f8-vdpb8"]
Oct 30 21:14:37.542545 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:37.542509    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8528cca8-1a08-4d6a-9291-3cf08da5f5c0" path="/var/lib/kubelet/pods/8528cca8-1a08-4d6a-9291-3cf08da5f5c0/volumes"
Oct 30 21:14:48.972406 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:48.972362    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"]
Oct 30 21:14:48.973139 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:48.973117    5967 cpu_manager.go:414] "containerMap: removing container" podUID="cfca4389-3d1a-46a5-9d17-7a540962de64" containerName="revoke-keys"
Oct 30 21:14:48.973231 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:48.973142    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfca4389-3d1a-46a5-9d17-7a540962de64" containerName="revoke-keys"
Oct 30 21:14:48.973291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:48.973276    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfca4389-3d1a-46a5-9d17-7a540962de64" containerName="revoke-keys"
Oct 30 21:14:48.978166 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:48.978140    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.030794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.030714    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:49.034076 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.034036    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"]
Oct 30 21:14:49.080375 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.080326    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:14:49.080575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.080382    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:49.080575 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.080452    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:49.080724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.080703    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.080915 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.080896    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpsx\" (UniqueName: \"kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.080967 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.080955    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.181706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.181667    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpsx\" (UniqueName: \"kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.181988 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.181732    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.181988 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.181943    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.182579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.182556    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.184408 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.184385    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.239873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.239798    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpsx\" (UniqueName: \"kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx\") pod \"maas-api-revoke-sub-79805ca9-dtjkg\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.293874 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.293836    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:49.667422 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:49.667396    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"]
Oct 30 21:14:49.669660 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:14:49.669631    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b61bad_ee86_4bd8_83ab_b7f8678c2ff0.slice/crio-c432e3e994c93d607e0a7f8d07a2b0331ea4eae89f3d04755c27b36fa71b6e8f WatchSource:0}: Error finding container c432e3e994c93d607e0a7f8d07a2b0331ea4eae89f3d04755c27b36fa71b6e8f: Status 404 returned error can't find the container with id c432e3e994c93d607e0a7f8d07a2b0331ea4eae89f3d04755c27b36fa71b6e8f
Oct 30 21:14:50.407333 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:50.407233    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg" event={"ID":"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0","Type":"ContainerStarted","Data":"c3475dcb44aa3cf43e2fb4b7f0195c46ac600c6390b451c6aa385242c47d6f4b"}
Oct 30 21:14:50.407333 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:50.407297    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg" event={"ID":"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0","Type":"ContainerStarted","Data":"c432e3e994c93d607e0a7f8d07a2b0331ea4eae89f3d04755c27b36fa71b6e8f"}
Oct 30 21:14:50.522209 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:50.522152    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg" podStartSLOduration=2.522136835 podStartE2EDuration="2.522136835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:50.520058563 +0000 UTC m=+3271.120860525" watchObservedRunningTime="2026-10-30 21:14:50.522136835 +0000 UTC m=+3271.122938792"
Oct 30 21:14:51.415123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:51.415089    5967 generic.go:356] "Generic (PLEG): container finished" podID="a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" containerID="c3475dcb44aa3cf43e2fb4b7f0195c46ac600c6390b451c6aa385242c47d6f4b" exitCode=0
Oct 30 21:14:51.415571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:51.415172    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg" event={"ID":"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0","Type":"ContainerDied","Data":"c3475dcb44aa3cf43e2fb4b7f0195c46ac600c6390b451c6aa385242c47d6f4b"}
Oct 30 21:14:52.557054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.557023    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:52.617184 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.617152    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx\" (UniqueName: \"kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx\") pod \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") "
Oct 30 21:14:52.617410 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.617271    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca\") pod \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") "
Oct 30 21:14:52.617410 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.617326    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca\") pod \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\" (UID: \"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0\") "
Oct 30 21:14:52.617621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.617597    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca" pod "a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" (UID: "a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:52.619480 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.619453    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca" pod "a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" (UID: "a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:52.619565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.619498    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx" pod "a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" (UID: "a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0"). InnerVolumeSpecName "kube-api-access-jzpsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:52.718318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.718285    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:52.718318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.718313    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:52.718318 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:52.718323    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzpsx\" (UniqueName: \"kubernetes.io/projected/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0-kube-api-access-jzpsx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:53.427317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:53.427279    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg" event={"ID":"a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0","Type":"ContainerDied","Data":"c432e3e994c93d607e0a7f8d07a2b0331ea4eae89f3d04755c27b36fa71b6e8f"}
Oct 30 21:14:53.427317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:53.427309    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"
Oct 30 21:14:53.427561 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:14:53.427332    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c432e3e994c93d607e0a7f8d07a2b0331ea4eae89f3d04755c27b36fa71b6e8f"
Oct 30 21:15:00.162663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.162629    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"]
Oct 30 21:15:00.163112 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.163102    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" containerName="revoke-keys"
Oct 30 21:15:00.163157 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.163113    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" containerName="revoke-keys"
Oct 30 21:15:00.163214 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.163203    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" containerName="revoke-keys"
Oct 30 21:15:00.167410 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.167391    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.223312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.223276    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"]
Oct 30 21:15:00.269985 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.269946    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:00.269985 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.269970    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:15:00.270247 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.269970    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:00.296439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.296403    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wwf\" (UniqueName: \"kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf\") pod \"maas-api-key-cleanup-29889915-4knpz\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.296635 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.296536    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-4knpz\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.397799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.397728    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wwf\" (UniqueName: \"kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf\") pod \"maas-api-key-cleanup-29889915-4knpz\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.398029 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.397866    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-4knpz\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.398510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.398490    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-4knpz\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.454155 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.454074    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wwf\" (UniqueName: \"kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf\") pod \"maas-api-key-cleanup-29889915-4knpz\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.496508 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.496476    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:00.871005 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:00.870975    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"]
Oct 30 21:15:00.874223 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:15:00.874194    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b530eb9_fb98_41b0_8ea7_8e32838845f4.slice/crio-6f28e1cb57fd96d310647b908cc3b46c7d820d84eabf22c09c1fbb2f6573c0d0 WatchSource:0}: Error finding container 6f28e1cb57fd96d310647b908cc3b46c7d820d84eabf22c09c1fbb2f6573c0d0: Status 404 returned error can't find the container with id 6f28e1cb57fd96d310647b908cc3b46c7d820d84eabf22c09c1fbb2f6573c0d0
Oct 30 21:15:01.473883 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:01.473843    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" event={"ID":"7b530eb9-fb98-41b0-8ea7-8e32838845f4","Type":"ContainerStarted","Data":"d9d31f127ec81a11399716ef7d1fabf8e20413b23d6edd8ad2daa001de850531"}
Oct 30 21:15:01.474374 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:01.473907    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" event={"ID":"7b530eb9-fb98-41b0-8ea7-8e32838845f4","Type":"ContainerStarted","Data":"6f28e1cb57fd96d310647b908cc3b46c7d820d84eabf22c09c1fbb2f6573c0d0"}
Oct 30 21:15:01.587837 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:01.587783    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" podStartSLOduration=1.587765726 podStartE2EDuration="1.587765726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.586147213 +0000 UTC m=+3282.186949176" watchObservedRunningTime="2026-10-30 21:15:01.587765726 +0000 UTC m=+3282.188567682"
Oct 30 21:15:01.980085 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:15:01.980048    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:15:02.480381 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:02.480343    5967 generic.go:356] "Generic (PLEG): container finished" podID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerID="d9d31f127ec81a11399716ef7d1fabf8e20413b23d6edd8ad2daa001de850531" exitCode=60
Oct 30 21:15:02.480925 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:02.480435    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" event={"ID":"7b530eb9-fb98-41b0-8ea7-8e32838845f4","Type":"ContainerDied","Data":"d9d31f127ec81a11399716ef7d1fabf8e20413b23d6edd8ad2daa001de850531"}
Oct 30 21:15:02.480925 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:02.480734    5967 scope.go:122] "RemoveContainer" containerID="d9d31f127ec81a11399716ef7d1fabf8e20413b23d6edd8ad2daa001de850531"
Oct 30 21:15:03.491097 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:03.491063    5967 generic.go:356] "Generic (PLEG): container finished" podID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerID="c9e0b369904bd964f62bc4c704b2cc02dd1f39e3787fff46c9983a6253b6f37a" exitCode=60
Oct 30 21:15:03.491597 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:03.491149    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" event={"ID":"7b530eb9-fb98-41b0-8ea7-8e32838845f4","Type":"ContainerDied","Data":"c9e0b369904bd964f62bc4c704b2cc02dd1f39e3787fff46c9983a6253b6f37a"}
Oct 30 21:15:03.491597 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:03.491205    5967 scope.go:122] "RemoveContainer" containerID="d9d31f127ec81a11399716ef7d1fabf8e20413b23d6edd8ad2daa001de850531"
Oct 30 21:15:03.491597 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:03.491477    5967 scope.go:122] "RemoveContainer" containerID="c9e0b369904bd964f62bc4c704b2cc02dd1f39e3787fff46c9983a6253b6f37a"
Oct 30 21:15:03.491850 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:15:03.491809    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-4knpz_odh-ai-gateway-infra(7b530eb9-fb98-41b0-8ea7-8e32838845f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4"
Oct 30 21:15:04.500512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:04.500482    5967 scope.go:122] "RemoveContainer" containerID="c9e0b369904bd964f62bc4c704b2cc02dd1f39e3787fff46c9983a6253b6f37a"
Oct 30 21:15:04.501057 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:15:04.500839    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-4knpz_odh-ai-gateway-infra(7b530eb9-fb98-41b0-8ea7-8e32838845f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4"
Oct 30 21:15:16.536371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:16.536334    5967 scope.go:122] "RemoveContainer" containerID="c9e0b369904bd964f62bc4c704b2cc02dd1f39e3787fff46c9983a6253b6f37a"
Oct 30 21:15:17.592168 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:17.592134    5967 generic.go:356] "Generic (PLEG): container finished" podID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerID="75714c014ad59f0c8076eb4b5a23a06e37ab5d0a690a54fcced098235cb8d5c2" exitCode=60
Oct 30 21:15:17.592692 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:17.592175    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" event={"ID":"7b530eb9-fb98-41b0-8ea7-8e32838845f4","Type":"ContainerDied","Data":"75714c014ad59f0c8076eb4b5a23a06e37ab5d0a690a54fcced098235cb8d5c2"}
Oct 30 21:15:17.592692 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:17.592243    5967 scope.go:122] "RemoveContainer" containerID="c9e0b369904bd964f62bc4c704b2cc02dd1f39e3787fff46c9983a6253b6f37a"
Oct 30 21:15:17.592878 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:17.592723    5967 scope.go:122] "RemoveContainer" containerID="75714c014ad59f0c8076eb4b5a23a06e37ab5d0a690a54fcced098235cb8d5c2"
Oct 30 21:15:17.593090 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:15:17.593067    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-4knpz_odh-ai-gateway-infra(7b530eb9-fb98-41b0-8ea7-8e32838845f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4"
Oct 30 21:15:18.720954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:18.720910    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"]
Oct 30 21:15:18.878364 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:18.878337    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:19.000583 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.000504    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca\") pod \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") "
Oct 30 21:15:19.000583 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.000557    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf\" (UniqueName: \"kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf\") pod \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\" (UID: \"7b530eb9-fb98-41b0-8ea7-8e32838845f4\") "
Oct 30 21:15:19.000999 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.000975    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca" pod "7b530eb9-fb98-41b0-8ea7-8e32838845f4" (UID: "7b530eb9-fb98-41b0-8ea7-8e32838845f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.002952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.002925    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf" pod "7b530eb9-fb98-41b0-8ea7-8e32838845f4" (UID: "7b530eb9-fb98-41b0-8ea7-8e32838845f4"). InnerVolumeSpecName "kube-api-access-h2wwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.101714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.101676    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b530eb9-fb98-41b0-8ea7-8e32838845f4-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.101714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.101711    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wwf\" (UniqueName: \"kubernetes.io/projected/7b530eb9-fb98-41b0-8ea7-8e32838845f4-kube-api-access-h2wwf\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.606400 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.606369    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"
Oct 30 21:15:19.606573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.606369    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz" event={"ID":"7b530eb9-fb98-41b0-8ea7-8e32838845f4","Type":"ContainerDied","Data":"6f28e1cb57fd96d310647b908cc3b46c7d820d84eabf22c09c1fbb2f6573c0d0"}
Oct 30 21:15:19.606573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.606497    5967 scope.go:122] "RemoveContainer" containerID="75714c014ad59f0c8076eb4b5a23a06e37ab5d0a690a54fcced098235cb8d5c2"
Oct 30 21:15:19.769349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.769319    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"]
Oct 30 21:15:19.818361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:19.818274    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-4knpz"]
Oct 30 21:15:21.541718 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:21.541685    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" path="/var/lib/kubelet/pods/7b530eb9-fb98-41b0-8ea7-8e32838845f4/volumes"
Oct 30 21:15:22.817823 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:22.817786    5967 scope.go:122] "RemoveContainer" containerID="6340e159af254aa220ffd2c9e6536071b44785b9dd245cbd99526dbf23cf6a57"
Oct 30 21:15:22.833059 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:22.833034    5967 scope.go:122] "RemoveContainer" containerID="92075d8140db68de9165c1c2d470923306f626886c600ac046c8faa0e5bb86a1"
Oct 30 21:15:24.335444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.335405    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"]
Oct 30 21:15:24.335965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.335950    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:24.336017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.335967    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:24.336017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.335981    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:24.336017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.335985    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:24.336189 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.336075    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:24.336189 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.336089    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:24.340994 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.340974    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.393512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.393478    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:24.443029 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.442995    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:15:24.443029 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.442995    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:24.443259 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.442995    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:24.445279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.445256    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"]
Oct 30 21:15:24.456780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.456716    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.456952 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.456887    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmjx\" (UniqueName: \"kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.457007 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.456965    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.558377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.558334    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmjx\" (UniqueName: \"kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.558576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.558416    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.558576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.558511    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.559273 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.559183    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.560984 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.560960    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.614762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.614674    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmjx\" (UniqueName: \"kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx\") pod \"maas-api-revoke-sub-3d4156f1-l48jw\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:24.654851 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:24.654815    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:25.028955 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:25.028924    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"]
Oct 30 21:15:25.032944 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:15:25.032914    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222b02c7_bbdb_4344_ac19_3ed19e08e1e2.slice/crio-0047c2f9ac1c604b29311007f06f9d42a11f74e9b86389f37dca6c64a44d9f98 WatchSource:0}: Error finding container 0047c2f9ac1c604b29311007f06f9d42a11f74e9b86389f37dca6c64a44d9f98: Status 404 returned error can't find the container with id 0047c2f9ac1c604b29311007f06f9d42a11f74e9b86389f37dca6c64a44d9f98
Oct 30 21:15:25.643339 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:25.643238    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw" event={"ID":"222b02c7-bbdb-4344-ac19-3ed19e08e1e2","Type":"ContainerStarted","Data":"db1d4f7320326b300d7d16b0c7ccedee8a3e1ee849fe77427d9d57d690e18609"}
Oct 30 21:15:25.643339 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:25.643295    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw" event={"ID":"222b02c7-bbdb-4344-ac19-3ed19e08e1e2","Type":"ContainerStarted","Data":"0047c2f9ac1c604b29311007f06f9d42a11f74e9b86389f37dca6c64a44d9f98"}
Oct 30 21:15:25.779069 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:25.779016    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw" podStartSLOduration=1.779002374 podStartE2EDuration="1.779002374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:25.77731261 +0000 UTC m=+3306.378114584" watchObservedRunningTime="2026-10-30 21:15:25.779002374 +0000 UTC m=+3306.379804334"
Oct 30 21:15:26.654442 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:26.654407    5967 generic.go:356] "Generic (PLEG): container finished" podID="222b02c7-bbdb-4344-ac19-3ed19e08e1e2" containerID="db1d4f7320326b300d7d16b0c7ccedee8a3e1ee849fe77427d9d57d690e18609" exitCode=0
Oct 30 21:15:26.654909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:26.654498    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw" event={"ID":"222b02c7-bbdb-4344-ac19-3ed19e08e1e2","Type":"ContainerDied","Data":"db1d4f7320326b300d7d16b0c7ccedee8a3e1ee849fe77427d9d57d690e18609"}
Oct 30 21:15:27.917066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.916983    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:27.939100 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.939067    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx\" (UniqueName: \"kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx\") pod \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") "
Oct 30 21:15:27.939266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.939150    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca\") pod \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") "
Oct 30 21:15:27.939266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.939250    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca\") pod \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\" (UID: \"222b02c7-bbdb-4344-ac19-3ed19e08e1e2\") "
Oct 30 21:15:27.939545 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.939518    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca" pod "222b02c7-bbdb-4344-ac19-3ed19e08e1e2" (UID: "222b02c7-bbdb-4344-ac19-3ed19e08e1e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:27.939662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.939607    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:27.941397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.941374    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx" pod "222b02c7-bbdb-4344-ac19-3ed19e08e1e2" (UID: "222b02c7-bbdb-4344-ac19-3ed19e08e1e2"). InnerVolumeSpecName "kube-api-access-mqmjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:27.941484 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:27.941411    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca" pod "222b02c7-bbdb-4344-ac19-3ed19e08e1e2" (UID: "222b02c7-bbdb-4344-ac19-3ed19e08e1e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:28.040840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:28.040792    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqmjx\" (UniqueName: \"kubernetes.io/projected/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-kube-api-access-mqmjx\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:28.040840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:28.040837    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222b02c7-bbdb-4344-ac19-3ed19e08e1e2-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:28.674978 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:28.674943    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"
Oct 30 21:15:28.674978 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:28.674971    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw" event={"ID":"222b02c7-bbdb-4344-ac19-3ed19e08e1e2","Type":"ContainerDied","Data":"0047c2f9ac1c604b29311007f06f9d42a11f74e9b86389f37dca6c64a44d9f98"}
Oct 30 21:15:28.675216 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:28.675048    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0047c2f9ac1c604b29311007f06f9d42a11f74e9b86389f37dca6c64a44d9f98"
Oct 30 21:15:31.981252 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:15:31.981215    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:15:33.045950 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:33.045913    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"]
Oct 30 21:15:33.095290 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:33.095253    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vk9t2"]
Oct 30 21:15:33.541639 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:33.541609    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08638f3-fdd4-4632-950a-3295822f330c" path="/var/lib/kubelet/pods/e08638f3-fdd4-4632-950a-3295822f330c/volumes"
Oct 30 21:15:55.164129 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.164094    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"]
Oct 30 21:15:55.214891 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.214859    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l48jw"]
Oct 30 21:15:55.285913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.285881    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"]
Oct 30 21:15:55.287115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.287085    5967 cpu_manager.go:414] "containerMap: removing container" podUID="222b02c7-bbdb-4344-ac19-3ed19e08e1e2" containerName="revoke-keys"
Oct 30 21:15:55.287115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.287115    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222b02c7-bbdb-4344-ac19-3ed19e08e1e2" containerName="revoke-keys"
Oct 30 21:15:55.287335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.287183    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:55.287335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.287191    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:55.287509 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.287494    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="222b02c7-bbdb-4344-ac19-3ed19e08e1e2" containerName="revoke-keys"
Oct 30 21:15:55.287569 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.287523    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b530eb9-fb98-41b0-8ea7-8e32838845f4" containerName="cleanup"
Oct 30 21:15:55.292131 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.292108    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.346783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.346718    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:55.347218 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.346729    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:55.347327 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.346792    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:55.347407 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.346904    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:15:55.349782 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.349761    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"]
Oct 30 21:15:55.378444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.378401    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.378444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.378451    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.378689 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.378575    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zfz\" (UniqueName: \"kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.479626 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.479595    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.479871 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.479646    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zfz\" (UniqueName: \"kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.479921 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.479866    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.480359 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.480337    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.482402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.482377    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.536117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.536090    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zfz\" (UniqueName: \"kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz\") pod \"maas-api-revoke-sub-3d4156f1-72xxb\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.542550 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.542522    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222b02c7-bbdb-4344-ac19-3ed19e08e1e2" path="/var/lib/kubelet/pods/222b02c7-bbdb-4344-ac19-3ed19e08e1e2/volumes"
Oct 30 21:15:55.608336 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.608305    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:55.984710 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:55.984675    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"]
Oct 30 21:15:55.986558 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:15:55.986528    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bb74b2_f534_44ea_8d20_c3e783a01dec.slice/crio-d7389621d0067a2aa4d1392e1e59513a3831609444b0bb0a505a4e961eeba6c9 WatchSource:0}: Error finding container d7389621d0067a2aa4d1392e1e59513a3831609444b0bb0a505a4e961eeba6c9: Status 404 returned error can't find the container with id d7389621d0067a2aa4d1392e1e59513a3831609444b0bb0a505a4e961eeba6c9
Oct 30 21:15:56.839538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:56.839501    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb" event={"ID":"c2bb74b2-f534-44ea-8d20-c3e783a01dec","Type":"ContainerStarted","Data":"90dc6bf68f70991dcff24cea37c3a5f55033047ee4da4824719c91014da56564"}
Oct 30 21:15:56.840073 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:56.839570    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb" event={"ID":"c2bb74b2-f534-44ea-8d20-c3e783a01dec","Type":"ContainerStarted","Data":"d7389621d0067a2aa4d1392e1e59513a3831609444b0bb0a505a4e961eeba6c9"}
Oct 30 21:15:56.952869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:56.952792    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb" podStartSLOduration=1.952774314 podStartE2EDuration="1.952774314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:56.95217008 +0000 UTC m=+3337.552972038" watchObservedRunningTime="2026-10-30 21:15:56.952774314 +0000 UTC m=+3337.553576255"
Oct 30 21:15:57.847649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:57.847612    5967 generic.go:356] "Generic (PLEG): container finished" podID="c2bb74b2-f534-44ea-8d20-c3e783a01dec" containerID="90dc6bf68f70991dcff24cea37c3a5f55033047ee4da4824719c91014da56564" exitCode=0
Oct 30 21:15:57.848071 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:57.847701    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb" event={"ID":"c2bb74b2-f534-44ea-8d20-c3e783a01dec","Type":"ContainerDied","Data":"90dc6bf68f70991dcff24cea37c3a5f55033047ee4da4824719c91014da56564"}
Oct 30 21:15:58.996599 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:58.996575    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:59.117717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.117620    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca\") pod \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") "
Oct 30 21:15:59.117717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.117696    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca\") pod \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") "
Oct 30 21:15:59.118024 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.117832    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz\" (UniqueName: \"kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz\") pod \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\" (UID: \"c2bb74b2-f534-44ea-8d20-c3e783a01dec\") "
Oct 30 21:15:59.118117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.118094    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca" pod "c2bb74b2-f534-44ea-8d20-c3e783a01dec" (UID: "c2bb74b2-f534-44ea-8d20-c3e783a01dec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:59.119982 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.119958    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca" pod "c2bb74b2-f534-44ea-8d20-c3e783a01dec" (UID: "c2bb74b2-f534-44ea-8d20-c3e783a01dec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:59.120147 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.120133    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz" pod "c2bb74b2-f534-44ea-8d20-c3e783a01dec" (UID: "c2bb74b2-f534-44ea-8d20-c3e783a01dec"). InnerVolumeSpecName "kube-api-access-l6zfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:59.218663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.218629    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zfz\" (UniqueName: \"kubernetes.io/projected/c2bb74b2-f534-44ea-8d20-c3e783a01dec-kube-api-access-l6zfz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:59.218663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.218659    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:59.218663 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.218668    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bb74b2-f534-44ea-8d20-c3e783a01dec-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:59.861818 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.861705    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"
Oct 30 21:15:59.861818 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.861698    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb" event={"ID":"c2bb74b2-f534-44ea-8d20-c3e783a01dec","Type":"ContainerDied","Data":"d7389621d0067a2aa4d1392e1e59513a3831609444b0bb0a505a4e961eeba6c9"}
Oct 30 21:15:59.862041 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:15:59.861851    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7389621d0067a2aa4d1392e1e59513a3831609444b0bb0a505a4e961eeba6c9"
Oct 30 21:16:01.980965 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:16:01.980927    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:16:22.863791 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:22.863726    5967 scope.go:122] "RemoveContainer" containerID="e4349c03dd757f2b3aa68ab6e2846a60283f5fb964c0e2c4878c3c3d4cf8fcb3"
Oct 30 21:16:31.982136 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:16:31.982101    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:16:42.050925 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.050839    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"]
Oct 30 21:16:42.101580 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.101528    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xv524"]
Oct 30 21:16:42.971919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.971883    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"]
Oct 30 21:16:42.972434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.972420    5967 cpu_manager.go:414] "containerMap: removing container" podUID="c2bb74b2-f534-44ea-8d20-c3e783a01dec" containerName="revoke-keys"
Oct 30 21:16:42.972483 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.972436    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bb74b2-f534-44ea-8d20-c3e783a01dec" containerName="revoke-keys"
Oct 30 21:16:42.972536 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.972527    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bb74b2-f534-44ea-8d20-c3e783a01dec" containerName="revoke-keys"
Oct 30 21:16:42.977049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:42.977029    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.080274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.080239    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:43.080274 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.080280    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:43.080726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.080313    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:16:43.080726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.080338    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:43.082639 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.082612    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"]
Oct 30 21:16:43.100358 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.100329    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tph8\" (UniqueName: \"kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.100519 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.100397    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.100574 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.100515    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.201345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.201304    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tph8\" (UniqueName: \"kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.201567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.201375    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.201567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.201444    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.202236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.202141    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.204075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.204050    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.258135 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.258056    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tph8\" (UniqueName: \"kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8\") pod \"maas-api-revoke-sub-e2fc9c5b-9jxq8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.290611 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.290580    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:43.542700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.542619    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29408351-8883-4227-b164-9b0f06bf9e50" path="/var/lib/kubelet/pods/29408351-8883-4227-b164-9b0f06bf9e50/volumes"
Oct 30 21:16:43.667629 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:43.667605    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"]
Oct 30 21:16:44.110204 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:44.110172    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8" event={"ID":"033000d1-1f7f-4038-89c8-70b9f44866e8","Type":"ContainerStarted","Data":"333a7ef395d67c2d3316ed9f967d25233c00d23a901e0a92144d6582dc1580d4"}
Oct 30 21:16:44.110763 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:44.110230    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8" event={"ID":"033000d1-1f7f-4038-89c8-70b9f44866e8","Type":"ContainerStarted","Data":"f68f29ce6341dbdd1546a21edc9af8d912d0fd3cfb54b905504c936a6f3a6ae0"}
Oct 30 21:16:44.230117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:44.230060    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8" podStartSLOduration=2.230043642 podStartE2EDuration="2.230043642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:44.22919417 +0000 UTC m=+3384.829996130" watchObservedRunningTime="2026-10-30 21:16:44.230043642 +0000 UTC m=+3384.830845603"
Oct 30 21:16:45.116845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:45.116806    5967 generic.go:356] "Generic (PLEG): container finished" podID="033000d1-1f7f-4038-89c8-70b9f44866e8" containerID="333a7ef395d67c2d3316ed9f967d25233c00d23a901e0a92144d6582dc1580d4" exitCode=0
Oct 30 21:16:45.117272 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:45.116865    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8" event={"ID":"033000d1-1f7f-4038-89c8-70b9f44866e8","Type":"ContainerDied","Data":"333a7ef395d67c2d3316ed9f967d25233c00d23a901e0a92144d6582dc1580d4"}
Oct 30 21:16:46.266876 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.266850    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:46.334468 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.334433    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca\") pod \"033000d1-1f7f-4038-89c8-70b9f44866e8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") "
Oct 30 21:16:46.334468 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.334474    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8\" (UniqueName: \"kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8\") pod \"033000d1-1f7f-4038-89c8-70b9f44866e8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") "
Oct 30 21:16:46.334707 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.334527    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca\") pod \"033000d1-1f7f-4038-89c8-70b9f44866e8\" (UID: \"033000d1-1f7f-4038-89c8-70b9f44866e8\") "
Oct 30 21:16:46.334937 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.334911    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca" pod "033000d1-1f7f-4038-89c8-70b9f44866e8" (UID: "033000d1-1f7f-4038-89c8-70b9f44866e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:46.336915 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.336892    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca" pod "033000d1-1f7f-4038-89c8-70b9f44866e8" (UID: "033000d1-1f7f-4038-89c8-70b9f44866e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:46.336915 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.336904    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8" pod "033000d1-1f7f-4038-89c8-70b9f44866e8" (UID: "033000d1-1f7f-4038-89c8-70b9f44866e8"). InnerVolumeSpecName "kube-api-access-5tph8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:46.435969 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.435872    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:46.435969 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.435918    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033000d1-1f7f-4038-89c8-70b9f44866e8-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:46.435969 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:46.435929    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tph8\" (UniqueName: \"kubernetes.io/projected/033000d1-1f7f-4038-89c8-70b9f44866e8-kube-api-access-5tph8\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:47.130140 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:47.130109    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"
Oct 30 21:16:47.130342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:47.130138    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8" event={"ID":"033000d1-1f7f-4038-89c8-70b9f44866e8","Type":"ContainerDied","Data":"f68f29ce6341dbdd1546a21edc9af8d912d0fd3cfb54b905504c936a6f3a6ae0"}
Oct 30 21:16:47.130342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:47.130189    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f68f29ce6341dbdd1546a21edc9af8d912d0fd3cfb54b905504c936a6f3a6ae0"
Oct 30 21:16:53.798197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.798156    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"]
Oct 30 21:16:53.798885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.798861    5967 cpu_manager.go:414] "containerMap: removing container" podUID="033000d1-1f7f-4038-89c8-70b9f44866e8" containerName="revoke-keys"
Oct 30 21:16:53.799009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.798897    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033000d1-1f7f-4038-89c8-70b9f44866e8" containerName="revoke-keys"
Oct 30 21:16:53.799073 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.799007    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="033000d1-1f7f-4038-89c8-70b9f44866e8" containerName="revoke-keys"
Oct 30 21:16:53.802317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.802291    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:53.859611 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.859570    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"]
Oct 30 21:16:53.905735 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.905702    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:53.906212 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.906181    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:16:53.906426 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.906267    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:53.906524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.906303    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:53.914714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.914676    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:53.914938 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.914724    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:53.914991 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:53.914968    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rft4b\" (UniqueName: \"kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.016445 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.016406    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.016445 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.016450    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.016721 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.016482    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rft4b\" (UniqueName: \"kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.017173 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.017153    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.019106 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.019083    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.073854 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.073761    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rft4b\" (UniqueName: \"kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b\") pod \"maas-api-revoke-sub-5a29efb3-jrzxz\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.121462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.121425    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:54.500700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:54.500674    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"]
Oct 30 21:16:54.502544 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:16:54.502503    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee7429b_b2e2_4607_8d7f_da1da49d9fc3.slice/crio-75a1d1819e0c589a39af6542c8bae8343ffb57fdbb0ed9a30b50b6e2fd8fbb03 WatchSource:0}: Error finding container 75a1d1819e0c589a39af6542c8bae8343ffb57fdbb0ed9a30b50b6e2fd8fbb03: Status 404 returned error can't find the container with id 75a1d1819e0c589a39af6542c8bae8343ffb57fdbb0ed9a30b50b6e2fd8fbb03
Oct 30 21:16:55.177961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:55.177665    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz" event={"ID":"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3","Type":"ContainerStarted","Data":"a2fe207c1f3e33386ac3f9ebdf8f4a61b730d9c81172db3f8bc08b2d3fd2f9a0"}
Oct 30 21:16:55.177961 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:55.177726    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz" event={"ID":"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3","Type":"ContainerStarted","Data":"75a1d1819e0c589a39af6542c8bae8343ffb57fdbb0ed9a30b50b6e2fd8fbb03"}
Oct 30 21:16:55.292732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:55.292677    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz" podStartSLOduration=2.292661841 podStartE2EDuration="2.292661841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:55.291156442 +0000 UTC m=+3395.891958402" watchObservedRunningTime="2026-10-30 21:16:55.292661841 +0000 UTC m=+3395.893463815"
Oct 30 21:16:56.184983 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:56.184946    5967 generic.go:356] "Generic (PLEG): container finished" podID="6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" containerID="a2fe207c1f3e33386ac3f9ebdf8f4a61b730d9c81172db3f8bc08b2d3fd2f9a0" exitCode=0
Oct 30 21:16:56.185427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:56.185045    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz" event={"ID":"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3","Type":"ContainerDied","Data":"a2fe207c1f3e33386ac3f9ebdf8f4a61b730d9c81172db3f8bc08b2d3fd2f9a0"}
Oct 30 21:16:57.651700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.651346    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:16:57.756608 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.756513    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca\") pod \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") "
Oct 30 21:16:57.756857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.756625    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b\" (UniqueName: \"kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b\") pod \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") "
Oct 30 21:16:57.756857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.756686    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca\") pod \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\" (UID: \"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3\") "
Oct 30 21:16:57.757017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.756943    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca" pod "6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" (UID: "6ee7429b-b2e2-4607-8d7f-da1da49d9fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:57.759012 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.758980    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b" pod "6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" (UID: "6ee7429b-b2e2-4607-8d7f-da1da49d9fc3"). InnerVolumeSpecName "kube-api-access-rft4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:57.759124 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.759017    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca" pod "6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" (UID: "6ee7429b-b2e2-4607-8d7f-da1da49d9fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:57.857594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.857489    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:57.857594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.857520    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rft4b\" (UniqueName: \"kubernetes.io/projected/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-kube-api-access-rft4b\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:57.857594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:57.857530    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.235223 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:58.235181    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz" event={"ID":"6ee7429b-b2e2-4607-8d7f-da1da49d9fc3","Type":"ContainerDied","Data":"75a1d1819e0c589a39af6542c8bae8343ffb57fdbb0ed9a30b50b6e2fd8fbb03"}
Oct 30 21:16:58.235592 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:58.235571    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a1d1819e0c589a39af6542c8bae8343ffb57fdbb0ed9a30b50b6e2fd8fbb03"
Oct 30 21:16:58.235804 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:16:58.235508    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"
Oct 30 21:17:01.982262 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:17:01.982223    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:17:14.048153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.048112    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"]
Oct 30 21:17:14.097732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.097685    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xhp4v"]
Oct 30 21:17:14.312052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.311969    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"]
Oct 30 21:17:14.313839 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.313806    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-72xxb"]
Oct 30 21:17:14.439949 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.439909    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"]
Oct 30 21:17:14.440447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.440434    5967 cpu_manager.go:414] "containerMap: removing container" podUID="6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" containerName="revoke-keys"
Oct 30 21:17:14.440495 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.440449    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" containerName="revoke-keys"
Oct 30 21:17:14.440531 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.440510    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" containerName="revoke-keys"
Oct 30 21:17:14.444065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.444047    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.499085 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.499048    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:14.499085 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.499046    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:17:14.499085 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.499046    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:14.501364 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.501329    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:14.505145 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.505099    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"]
Oct 30 21:17:14.535270 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.535234    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkq6\" (UniqueName: \"kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.535491 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.535293    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.535539 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.535480    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.636870 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.636772    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.637043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.636894    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.637043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.636944    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkq6\" (UniqueName: \"kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.637555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.637526    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.639276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.639257    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.695275 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.695243    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkq6\" (UniqueName: \"kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6\") pod \"maas-api-revoke-sub-3d4156f1-shwtp\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:14.757629 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:14.757585    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:15.134111 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:15.134086    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"]
Oct 30 21:17:15.137146 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:17:15.137113    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026dc109_9b62_4048_8a3c_4ff60670d266.slice/crio-d3d89f37996fb6ec7da7b75ebe79c63cd0325e7b6c404f3967556380ac7a6ac1 WatchSource:0}: Error finding container d3d89f37996fb6ec7da7b75ebe79c63cd0325e7b6c404f3967556380ac7a6ac1: Status 404 returned error can't find the container with id d3d89f37996fb6ec7da7b75ebe79c63cd0325e7b6c404f3967556380ac7a6ac1
Oct 30 21:17:15.330378 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:15.330337    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp" event={"ID":"026dc109-9b62-4048-8a3c-4ff60670d266","Type":"ContainerStarted","Data":"d3d89f37996fb6ec7da7b75ebe79c63cd0325e7b6c404f3967556380ac7a6ac1"}
Oct 30 21:17:15.541950 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:15.541918    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d655e45-b9c0-4368-ab89-1e21cfc739b4" path="/var/lib/kubelet/pods/2d655e45-b9c0-4368-ab89-1e21cfc739b4/volumes"
Oct 30 21:17:15.542360 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:15.542347    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bb74b2-f534-44ea-8d20-c3e783a01dec" path="/var/lib/kubelet/pods/c2bb74b2-f534-44ea-8d20-c3e783a01dec/volumes"
Oct 30 21:17:16.337792 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:16.337724    5967 generic.go:356] "Generic (PLEG): container finished" podID="026dc109-9b62-4048-8a3c-4ff60670d266" containerID="9d836b50395e25b560f85707dcc61fa87da071cd78ca576c35a4f0c8b33b5e44" exitCode=0
Oct 30 21:17:16.338209 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:16.337812    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp" event={"ID":"026dc109-9b62-4048-8a3c-4ff60670d266","Type":"ContainerDied","Data":"9d836b50395e25b560f85707dcc61fa87da071cd78ca576c35a4f0c8b33b5e44"}
Oct 30 21:17:17.487329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.487293    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:17.568600 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.568566    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca\") pod \"026dc109-9b62-4048-8a3c-4ff60670d266\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") "
Oct 30 21:17:17.568862 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.568630    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca\") pod \"026dc109-9b62-4048-8a3c-4ff60670d266\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") "
Oct 30 21:17:17.568862 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.568728    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6\" (UniqueName: \"kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6\") pod \"026dc109-9b62-4048-8a3c-4ff60670d266\" (UID: \"026dc109-9b62-4048-8a3c-4ff60670d266\") "
Oct 30 21:17:17.569010 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.568982    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca" pod "026dc109-9b62-4048-8a3c-4ff60670d266" (UID: "026dc109-9b62-4048-8a3c-4ff60670d266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:17.569246 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.569222    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:17.571075 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.571047    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca" pod "026dc109-9b62-4048-8a3c-4ff60670d266" (UID: "026dc109-9b62-4048-8a3c-4ff60670d266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:17.571175 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.571088    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6" pod "026dc109-9b62-4048-8a3c-4ff60670d266" (UID: "026dc109-9b62-4048-8a3c-4ff60670d266"). InnerVolumeSpecName "kube-api-access-bfkq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:17.670634 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.670531    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026dc109-9b62-4048-8a3c-4ff60670d266-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:17.670634 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:17.670565    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkq6\" (UniqueName: \"kubernetes.io/projected/026dc109-9b62-4048-8a3c-4ff60670d266-kube-api-access-bfkq6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:18.351555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:18.351523    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"
Oct 30 21:17:18.351815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:18.351526    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp" event={"ID":"026dc109-9b62-4048-8a3c-4ff60670d266","Type":"ContainerDied","Data":"d3d89f37996fb6ec7da7b75ebe79c63cd0325e7b6c404f3967556380ac7a6ac1"}
Oct 30 21:17:18.351815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:18.351673    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3d89f37996fb6ec7da7b75ebe79c63cd0325e7b6c404f3967556380ac7a6ac1"
Oct 30 21:17:22.889929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:22.889894    5967 scope.go:122] "RemoveContainer" containerID="c9e58d736ff06b2996e7ea9f837959c59214e1a10fd0edf0f786d1127ea7d301"
Oct 30 21:17:27.044798 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:27.044754    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"]
Oct 30 21:17:27.094296 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:27.094254    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-f5zmr"]
Oct 30 21:17:27.541550 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:27.541514    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6aaaed-57e1-4cf2-ada8-f510194c4359" path="/var/lib/kubelet/pods/3b6aaaed-57e1-4cf2-ada8-f510194c4359/volumes"
Oct 30 21:17:31.981806 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:17:31.981710    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:17:47.046877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.046831    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"]
Oct 30 21:17:47.047445 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.047316    5967 cpu_manager.go:414] "containerMap: removing container" podUID="026dc109-9b62-4048-8a3c-4ff60670d266" containerName="revoke-keys"
Oct 30 21:17:47.047445 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.047327    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026dc109-9b62-4048-8a3c-4ff60670d266" containerName="revoke-keys"
Oct 30 21:17:47.047445 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.047433    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="026dc109-9b62-4048-8a3c-4ff60670d266" containerName="revoke-keys"
Oct 30 21:17:47.052112 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.052093    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.103724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.103690    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-819d1b-openshift-default-dockercfg-72r9x\""
Oct 30 21:17:47.118489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118440    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118491    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118800 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118602    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118800 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118685    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118800 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118714    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkss\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118832    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118905    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.118941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.118932    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.119047 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.119018    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.155684 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.155632    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"]
Oct 30 21:17:47.220462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220427    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220468    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkss\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220511    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220546    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220571    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220601    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220661    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220687    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.220801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.220727    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.221312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.221042    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.221312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.221092    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.221312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.221134    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.221312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.221224    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.221538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.221392    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.223079 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.223055    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.223504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.223489    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.278565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.278529    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkss\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.278777 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.278730    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token\") pod \"e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") " pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.366003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.365923    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:47.803927 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.803894    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"]
Oct 30 21:17:47.805885 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:17:47.805853    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268f598a_6473_4f25_9b10_041d68adb13c.slice/crio-4e316c9d5a591d479d3f594d30c44e62bddc23f9fdbaed2a11d913f41d2b9862 WatchSource:0}: Error finding container 4e316c9d5a591d479d3f594d30c44e62bddc23f9fdbaed2a11d913f41d2b9862: Status 404 returned error can't find the container with id 4e316c9d5a591d479d3f594d30c44e62bddc23f9fdbaed2a11d913f41d2b9862
Oct 30 21:17:47.808215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.808181    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:47.808282 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.808249    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:47.808282 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:47.808276    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:48.547086 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:48.547040    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" event={"ID":"268f598a-6473-4f25-9b10-041d68adb13c","Type":"ContainerStarted","Data":"58affc23eefae4d079780e1a24761e2c7f3c2bac18973679366840683cc5571d"}
Oct 30 21:17:48.547511 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:48.547102    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" event={"ID":"268f598a-6473-4f25-9b10-041d68adb13c","Type":"ContainerStarted","Data":"4e316c9d5a591d479d3f594d30c44e62bddc23f9fdbaed2a11d913f41d2b9862"}
Oct 30 21:17:49.366933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:49.366892    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:49.372079 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:49.372052    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:49.539598 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:49.539529    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podStartSLOduration=2.539506294 podStartE2EDuration="2.539506294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:48.764888739 +0000 UTC m=+3449.365690701" watchObservedRunningTime="2026-10-30 21:17:49.539506294 +0000 UTC m=+3450.140308272"
Oct 30 21:17:49.557043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:49.557012    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:17:49.558053 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:17:49.558029    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:18:01.981086 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:18:01.981041    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:18:05.044812 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:05.044780    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"]
Oct 30 21:18:05.093606 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:05.093571    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-smx8z"]
Oct 30 21:18:05.543114 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:05.543074    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405f232a-e86b-4089-b454-6060d6892dc3" path="/var/lib/kubelet/pods/405f232a-e86b-4089-b454-6060d6892dc3/volumes"
Oct 30 21:18:15.731179 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:15.731144    5967 generic.go:356] "Generic (PLEG): container finished" podID="268f598a-6473-4f25-9b10-041d68adb13c" containerID="58affc23eefae4d079780e1a24761e2c7f3c2bac18973679366840683cc5571d" exitCode=0
Oct 30 21:18:15.731685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:15.731220    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" event={"ID":"268f598a-6473-4f25-9b10-041d68adb13c","Type":"ContainerDied","Data":"58affc23eefae4d079780e1a24761e2c7f3c2bac18973679366840683cc5571d"}
Oct 30 21:18:15.731685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:15.731668    5967 scope.go:122] "RemoveContainer" containerID="58affc23eefae4d079780e1a24761e2c7f3c2bac18973679366840683cc5571d"
Oct 30 21:18:15.733115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:15.733082    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:15.733220 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:15.733170    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:15.733220 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:15.733203    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:16.044654 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:16.044573    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"]
Oct 30 21:18:16.048567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:16.048537    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-79tz7"]
Oct 30 21:18:16.738581 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:16.738545    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" event={"ID":"268f598a-6473-4f25-9b10-041d68adb13c","Type":"ContainerStarted","Data":"4bc9cc6de179ff4f51005a9147c309f17ad590adba53818c8f3f8713bd5b18d6"}
Oct 30 21:18:17.366976 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:17.366946    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:18:17.368335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:17.368301    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:18:17.368489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:17.368373    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 21:18:17.543592 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:17.543555    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6f59b4-c6da-411a-a4c4-b5821e03ceb8" path="/var/lib/kubelet/pods/eb6f59b4-c6da-411a-a4c4-b5821e03ceb8/volumes"
Oct 30 21:18:18.367049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:18.367015    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:18:18.367564 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:18.367069    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 21:18:19.367287 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:19.367242    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:18:19.367783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:19.367316    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 21:18:19.557513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:19.557479    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:18:19.994505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:19.994448    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"]
Oct 30 21:18:20.000360 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.000332    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.051729 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.051638    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"]
Oct 30 21:18:20.081051 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.080992    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-819d1b\"/\"simulator-single-main-819d1b-kserve-self-signed-certs\""
Oct 30 21:18:20.097851 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.097820    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-819d1b\"/\"openshift-service-ca.crt\""
Oct 30 21:18:20.098176 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.098161    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-819d1b\"/\"kube-root-ca.crt\""
Oct 30 21:18:20.098369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.098357    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-819d1b\"/\"default-dockercfg-2jn5w\""
Oct 30 21:18:20.118638 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.118609    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.118869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.118715    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.118869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.118831    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.119016 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.118924    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.119016 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.118948    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.119016 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.118979    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b47b\" (UniqueName: \"kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220277    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b47b\" (UniqueName: \"kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220350    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220406    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220524 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220481    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220656 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220552    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220656 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220587    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.220949 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220921    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.221054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.220952    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.221054 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.221008    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.223001 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.222973    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.223237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.223211    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.284839 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.282538    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b47b\" (UniqueName: \"kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b\") pod \"simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") " pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.314542 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.314497    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:20.367248 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.367216    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:18:20.367423 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.367266    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 21:18:20.723679 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.723649    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"]
Oct 30 21:18:20.725967 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:18:20.725941    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod792648ab_0a9b_47aa_975c_7d4fda797933.slice/crio-24a1c081aa6693cb0fea8543c53720c1950befc1e1eacd92455f95bd94ae2af0 WatchSource:0}: Error finding container 24a1c081aa6693cb0fea8543c53720c1950befc1e1eacd92455f95bd94ae2af0: Status 404 returned error can't find the container with id 24a1c081aa6693cb0fea8543c53720c1950befc1e1eacd92455f95bd94ae2af0
Oct 30 21:18:20.763234 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:20.763194    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" event={"ID":"792648ab-0a9b-47aa-975c-7d4fda797933","Type":"ContainerStarted","Data":"24a1c081aa6693cb0fea8543c53720c1950befc1e1eacd92455f95bd94ae2af0"}
Oct 30 21:18:21.367172 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:21.367124    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:18:21.367611 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:21.367189    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 21:18:21.771971 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:21.771938    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" event={"ID":"792648ab-0a9b-47aa-975c-7d4fda797933","Type":"ContainerStarted","Data":"30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5"}
Oct 30 21:18:21.889313 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:21.889252    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" podStartSLOduration=2.889235889 podStartE2EDuration="2.889235889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:21.886798259 +0000 UTC m=+3482.487600219" watchObservedRunningTime="2026-10-30 21:18:21.889235889 +0000 UTC m=+3482.490037850"
Oct 30 21:18:22.315639 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:22.315603    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:22.370491 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:22.370462    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:18:22.371423 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:22.371405    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:18:22.913630 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:22.913593    5967 scope.go:122] "RemoveContainer" containerID="8dc9d8458cddc0e24d6ac4b5896f77e45b580b3459e40010e10f66953df9ad44"
Oct 30 21:18:22.926958 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:22.926931    5967 scope.go:122] "RemoveContainer" containerID="e71523a4fac67cc729f9f12bb916fa02a28478b95e87e0b060548a0a67010d61"
Oct 30 21:18:30.315593 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:30.315557    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:30.328296 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:30.328271    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:30.833506 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:30.833464    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:18:31.981358 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:18:31.981317    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:18:54.058808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:54.058770    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"]
Oct 30 21:18:54.108143 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:54.108104    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lnmvd"]
Oct 30 21:18:55.541891 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:18:55.541853    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48cd39b-41fc-47c5-8fbd-16fd1008c261" path="/var/lib/kubelet/pods/c48cd39b-41fc-47c5-8fbd-16fd1008c261/volumes"
Oct 30 21:19:01.980811 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:19:01.980687    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:19:05.108163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.108129    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"]
Oct 30 21:19:05.111986 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.111967    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.166199 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.166161    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:05.204092 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.204054    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.204284 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.204225    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.204328 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.204316    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rb4\" (UniqueName: \"kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.215379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.215329    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:05.215379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.215359    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:05.215624 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.215421    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:19:05.217567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.217541    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"]
Oct 30 21:19:05.305818 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.305778    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rb4\" (UniqueName: \"kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.306040 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.305844    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.306040 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.305973    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.306614 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.306588    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.308352 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.308334    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.362438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.362362    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rb4\" (UniqueName: \"kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4\") pod \"maas-api-revoke-sub-47bda64f-c4pl7\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.425830 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.425794    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:05.803994 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:05.803967    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"]
Oct 30 21:19:05.806734 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:19:05.806707    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e92c6f8_d9fd_4055_a478_f4f646911a91.slice/crio-20eade7ad2585d3754b091991b7cd63b832ad15a1f23fd58633e7918e0f652d2 WatchSource:0}: Error finding container 20eade7ad2585d3754b091991b7cd63b832ad15a1f23fd58633e7918e0f652d2: Status 404 returned error can't find the container with id 20eade7ad2585d3754b091991b7cd63b832ad15a1f23fd58633e7918e0f652d2
Oct 30 21:19:06.014061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:06.014025    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7" event={"ID":"9e92c6f8-d9fd-4055-a478-f4f646911a91","Type":"ContainerStarted","Data":"20eade7ad2585d3754b091991b7cd63b832ad15a1f23fd58633e7918e0f652d2"}
Oct 30 21:19:07.021009 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:07.020977    5967 generic.go:356] "Generic (PLEG): container finished" podID="9e92c6f8-d9fd-4055-a478-f4f646911a91" containerID="dadc3930c0b7098372f1df1d821aeb670c3d453868b61c1b1b32a0847e543342" exitCode=0
Oct 30 21:19:07.021513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:07.021049    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7" event={"ID":"9e92c6f8-d9fd-4055-a478-f4f646911a91","Type":"ContainerDied","Data":"dadc3930c0b7098372f1df1d821aeb670c3d453868b61c1b1b32a0847e543342"}
Oct 30 21:19:08.171932 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.171905    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:08.233159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.233125    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca\") pod \"9e92c6f8-d9fd-4055-a478-f4f646911a91\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") "
Oct 30 21:19:08.233361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.233241    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca\") pod \"9e92c6f8-d9fd-4055-a478-f4f646911a91\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") "
Oct 30 21:19:08.233361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.233289    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4\" (UniqueName: \"kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4\") pod \"9e92c6f8-d9fd-4055-a478-f4f646911a91\" (UID: \"9e92c6f8-d9fd-4055-a478-f4f646911a91\") "
Oct 30 21:19:08.233626 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.233595    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca" pod "9e92c6f8-d9fd-4055-a478-f4f646911a91" (UID: "9e92c6f8-d9fd-4055-a478-f4f646911a91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:08.235430 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.235407    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca" pod "9e92c6f8-d9fd-4055-a478-f4f646911a91" (UID: "9e92c6f8-d9fd-4055-a478-f4f646911a91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:08.235567 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.235463    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4" pod "9e92c6f8-d9fd-4055-a478-f4f646911a91" (UID: "9e92c6f8-d9fd-4055-a478-f4f646911a91"). InnerVolumeSpecName "kube-api-access-n2rb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:08.335116 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.335032    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:08.335116 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.335060    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92c6f8-d9fd-4055-a478-f4f646911a91-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:08.335116 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:08.335072    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rb4\" (UniqueName: \"kubernetes.io/projected/9e92c6f8-d9fd-4055-a478-f4f646911a91-kube-api-access-n2rb4\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:09.033602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:09.033573    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"
Oct 30 21:19:09.033602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:09.033587    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7" event={"ID":"9e92c6f8-d9fd-4055-a478-f4f646911a91","Type":"ContainerDied","Data":"20eade7ad2585d3754b091991b7cd63b832ad15a1f23fd58633e7918e0f652d2"}
Oct 30 21:19:09.033845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:09.033631    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20eade7ad2585d3754b091991b7cd63b832ad15a1f23fd58633e7918e0f652d2"
Oct 30 21:19:17.270724 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.270688    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"]
Oct 30 21:19:17.271192 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.271179    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9e92c6f8-d9fd-4055-a478-f4f646911a91" containerName="revoke-keys"
Oct 30 21:19:17.271237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.271193    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e92c6f8-d9fd-4055-a478-f4f646911a91" containerName="revoke-keys"
Oct 30 21:19:17.271279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.271270    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e92c6f8-d9fd-4055-a478-f4f646911a91" containerName="revoke-keys"
Oct 30 21:19:17.274695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.274676    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.338324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.338289    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p2z\" (UniqueName: \"kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.338641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.338576    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.338780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.338673    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.378776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.378704    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:17.378776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.378733    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:17.379026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.378815    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:19:17.379026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.378898    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:17.381183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.381162    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"]
Oct 30 21:19:17.440393 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.440336    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p2z\" (UniqueName: \"kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.440909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.440506    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.440909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.440600    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.441474 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.441451    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.443357 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.443334    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.496997 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.496966    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p2z\" (UniqueName: \"kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.590678 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.590594    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:17.967105 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:17.967080    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"]
Oct 30 21:19:17.970032 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:19:17.969990    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf44fbff_961f_49d6_b42b_b2361609e4ac.slice/crio-f3b5c21e60edef16365f400c6519b5f86a607a5bef0a5ef6f9a7be7adb906c46 WatchSource:0}: Error finding container f3b5c21e60edef16365f400c6519b5f86a607a5bef0a5ef6f9a7be7adb906c46: Status 404 returned error can't find the container with id f3b5c21e60edef16365f400c6519b5f86a607a5bef0a5ef6f9a7be7adb906c46
Oct 30 21:19:18.085176 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:18.085142    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j" event={"ID":"df44fbff-961f-49d6-b42b-b2361609e4ac","Type":"ContainerStarted","Data":"f3b5c21e60edef16365f400c6519b5f86a607a5bef0a5ef6f9a7be7adb906c46"}
Oct 30 21:19:19.092914 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:19.092802    5967 generic.go:356] "Generic (PLEG): container finished" podID="df44fbff-961f-49d6-b42b-b2361609e4ac" containerID="4e9d4975fd32c856088dccb4e2e6b40b40ec78ab7d2508821b107d207dbd38d1" exitCode=0
Oct 30 21:19:19.093317 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:19.092904    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j" event={"ID":"df44fbff-961f-49d6-b42b-b2361609e4ac","Type":"ContainerDied","Data":"4e9d4975fd32c856088dccb4e2e6b40b40ec78ab7d2508821b107d207dbd38d1"}
Oct 30 21:19:20.252207 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.252170    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:20.376599 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.376515    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca\") pod \"df44fbff-961f-49d6-b42b-b2361609e4ac\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") "
Oct 30 21:19:20.376810 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.376676    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca\") pod \"df44fbff-961f-49d6-b42b-b2361609e4ac\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") "
Oct 30 21:19:20.376914 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.376804    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z\" (UniqueName: \"kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z\") pod \"df44fbff-961f-49d6-b42b-b2361609e4ac\" (UID: \"df44fbff-961f-49d6-b42b-b2361609e4ac\") "
Oct 30 21:19:20.377171 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.377137    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca" pod "df44fbff-961f-49d6-b42b-b2361609e4ac" (UID: "df44fbff-961f-49d6-b42b-b2361609e4ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:20.379060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.379025    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca" pod "df44fbff-961f-49d6-b42b-b2361609e4ac" (UID: "df44fbff-961f-49d6-b42b-b2361609e4ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:20.379622 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.379594    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z" pod "df44fbff-961f-49d6-b42b-b2361609e4ac" (UID: "df44fbff-961f-49d6-b42b-b2361609e4ac"). InnerVolumeSpecName "kube-api-access-98p2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:20.478644 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.478603    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:20.478644 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.478638    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98p2z\" (UniqueName: \"kubernetes.io/projected/df44fbff-961f-49d6-b42b-b2361609e4ac-kube-api-access-98p2z\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:20.478644 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:20.478648    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df44fbff-961f-49d6-b42b-b2361609e4ac-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:21.105576 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:21.105544    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"
Oct 30 21:19:21.105792 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:21.105537    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j" event={"ID":"df44fbff-961f-49d6-b42b-b2361609e4ac","Type":"ContainerDied","Data":"f3b5c21e60edef16365f400c6519b5f86a607a5bef0a5ef6f9a7be7adb906c46"}
Oct 30 21:19:21.105792 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:21.105668    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b5c21e60edef16365f400c6519b5f86a607a5bef0a5ef6f9a7be7adb906c46"
Oct 30 21:19:21.716420 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:21.716385    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"]
Oct 30 21:19:22.044911 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:22.044775    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"]
Oct 30 21:19:22.047086 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:22.047061    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q82r5"]
Oct 30 21:19:22.967602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:22.967571    5967 scope.go:122] "RemoveContainer" containerID="9bbec052d4589a3dc272859aedadea321f0ab98c75b5c780abbaf6a7bd335d55"
Oct 30 21:19:22.977342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:22.977319    5967 scope.go:122] "RemoveContainer" containerID="88846bd3d9d9a2c2ddece1785c53dfa3f497515327be13ce6ae5ac8f077ba184"
Oct 30 21:19:22.986288 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:22.986265    5967 scope.go:122] "RemoveContainer" containerID="238730b5d7cceb9b31b301df1db32330a867e9eab5a144c4e32bffc291db3f7d"
Oct 30 21:19:23.542019 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:23.541990    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfca4389-3d1a-46a5-9d17-7a540962de64" path="/var/lib/kubelet/pods/cfca4389-3d1a-46a5-9d17-7a540962de64/volumes"
Oct 30 21:19:31.980540 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:19:31.980497    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:19:36.737511 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:36.737440    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" podUID="792648ab-0a9b-47aa-975c-7d4fda797933" containerName="main" containerID="cri-o://30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5" gracePeriod=45
Oct 30 21:19:36.822898 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:36.822850    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" podUID="792648ab-0a9b-47aa-975c-7d4fda797933" containerName="main" probeResult="failure" output="Get \"https://10.134.0.252:8000/ready\": dial tcp 10.134.0.252:8000: connect: connection refused"
Oct 30 21:19:36.993250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:36.993183    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:19:37.056286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056246    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir\") pod \"792648ab-0a9b-47aa-975c-7d4fda797933\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") "
Oct 30 21:19:37.056286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056288    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache\") pod \"792648ab-0a9b-47aa-975c-7d4fda797933\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") "
Oct 30 21:19:37.056515 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056310    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home\") pod \"792648ab-0a9b-47aa-975c-7d4fda797933\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") "
Oct 30 21:19:37.056515 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056361    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs\" (UniqueName: \"kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs\") pod \"792648ab-0a9b-47aa-975c-7d4fda797933\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") "
Oct 30 21:19:37.056515 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056394    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b\" (UniqueName: \"kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b\") pod \"792648ab-0a9b-47aa-975c-7d4fda797933\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") "
Oct 30 21:19:37.056666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056570    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm\") pod \"792648ab-0a9b-47aa-975c-7d4fda797933\" (UID: \"792648ab-0a9b-47aa-975c-7d4fda797933\") "
Oct 30 21:19:37.056666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056584    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir" pod "792648ab-0a9b-47aa-975c-7d4fda797933" (UID: "792648ab-0a9b-47aa-975c-7d4fda797933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:37.056666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056600    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache" pod "792648ab-0a9b-47aa-975c-7d4fda797933" (UID: "792648ab-0a9b-47aa-975c-7d4fda797933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:37.056666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056620    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home" pod "792648ab-0a9b-47aa-975c-7d4fda797933" (UID: "792648ab-0a9b-47aa-975c-7d4fda797933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:37.056951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056878    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.056951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056892    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.056951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.056900    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.058855 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.058831    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm" pod "792648ab-0a9b-47aa-975c-7d4fda797933" (UID: "792648ab-0a9b-47aa-975c-7d4fda797933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:37.059003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.058873    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs" pod "792648ab-0a9b-47aa-975c-7d4fda797933" (UID: "792648ab-0a9b-47aa-975c-7d4fda797933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:37.059003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.058924    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b" pod "792648ab-0a9b-47aa-975c-7d4fda797933" (UID: "792648ab-0a9b-47aa-975c-7d4fda797933"). InnerVolumeSpecName "kube-api-access-4b47b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:37.158888 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.158849    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792648ab-0a9b-47aa-975c-7d4fda797933-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.158888 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.158889    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792648ab-0a9b-47aa-975c-7d4fda797933-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.159096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.158906    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b47b\" (UniqueName: \"kubernetes.io/projected/792648ab-0a9b-47aa-975c-7d4fda797933-kube-api-access-4b47b\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:37.194424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.194379    5967 generic.go:356] "Generic (PLEG): container finished" podID="792648ab-0a9b-47aa-975c-7d4fda797933" containerID="30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5" exitCode=0
Oct 30 21:19:37.194424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.194422    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" event={"ID":"792648ab-0a9b-47aa-975c-7d4fda797933","Type":"ContainerDied","Data":"30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5"}
Oct 30 21:19:37.194695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.194455    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"
Oct 30 21:19:37.194695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.194469    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9" event={"ID":"792648ab-0a9b-47aa-975c-7d4fda797933","Type":"ContainerDied","Data":"24a1c081aa6693cb0fea8543c53720c1950befc1e1eacd92455f95bd94ae2af0"}
Oct 30 21:19:37.194695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.194491    5967 scope.go:122] "RemoveContainer" containerID="30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5"
Oct 30 21:19:37.205489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.205468    5967 scope.go:122] "RemoveContainer" containerID="30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5"
Oct 30 21:19:37.205806 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:19:37.205785    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5\": container with ID starting with 30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5 not found: ID does not exist" containerID="30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5"
Oct 30 21:19:37.205902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.205813    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5"} err="failed to get container status \"30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5\": rpc error: code = NotFound desc = could not find container \"30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5\": container with ID starting with 30f791947f1bd232aead7b1780e81f27c5b641a45562229f5623a2f89d1bd2c5 not found: ID does not exist"
Oct 30 21:19:37.357603 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.357519    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"]
Oct 30 21:19:37.361308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.361281    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-819d1b/simulator-single-main-819d1b-kserve-58b8cfd7d7-xjqt9"]
Oct 30 21:19:37.546968 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.546934    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792648ab-0a9b-47aa-975c-7d4fda797933" path="/var/lib/kubelet/pods/792648ab-0a9b-47aa-975c-7d4fda797933/volumes"
Oct 30 21:19:37.547329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.547313    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"]
Oct 30 21:19:37.547377 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:37.547353    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-819d1b-5ll6j"]
Oct 30 21:19:39.543149 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:39.543115    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df44fbff-961f-49d6-b42b-b2361609e4ac" path="/var/lib/kubelet/pods/df44fbff-961f-49d6-b42b-b2361609e4ac/volumes"
Oct 30 21:19:44.137628 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:44.137594    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"]
Oct 30 21:19:44.138130 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:44.137829    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy" containerID="cri-o://4bc9cc6de179ff4f51005a9147c309f17ad590adba53818c8f3f8713bd5b18d6" gracePeriod=30
Oct 30 21:19:45.808221 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.808174    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"]
Oct 30 21:19:45.857345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.857305    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shwtp"]
Oct 30 21:19:45.930533 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.930490    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"]
Oct 30 21:19:45.931017 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.931003    5967 cpu_manager.go:414] "containerMap: removing container" podUID="792648ab-0a9b-47aa-975c-7d4fda797933" containerName="main"
Oct 30 21:19:45.931069 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.931017    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792648ab-0a9b-47aa-975c-7d4fda797933" containerName="main"
Oct 30 21:19:45.931069 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.931034    5967 cpu_manager.go:414] "containerMap: removing container" podUID="df44fbff-961f-49d6-b42b-b2361609e4ac" containerName="revoke-keys"
Oct 30 21:19:45.931069 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.931039    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df44fbff-961f-49d6-b42b-b2361609e4ac" containerName="revoke-keys"
Oct 30 21:19:45.931183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.931119    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="792648ab-0a9b-47aa-975c-7d4fda797933" containerName="main"
Oct 30 21:19:45.931183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.931129    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="df44fbff-961f-49d6-b42b-b2361609e4ac" containerName="revoke-keys"
Oct 30 21:19:45.935789 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.935766    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:45.989621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.989589    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:45.989920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.989824    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:19:45.989920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.989913    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:45.990147 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.989821    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:45.993167 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:45.993142    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"]
Oct 30 21:19:46.052280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.051887    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5x92\" (UniqueName: \"kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.052280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.052043    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.052280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.052110    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.153334 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.153248    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5x92\" (UniqueName: \"kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.153334 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.153332    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.153556 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.153372    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.154117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.154095    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.156071 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.156049    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.211566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.211526    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5x92\" (UniqueName: \"kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92\") pod \"maas-api-revoke-sub-3d4156f1-cnmpk\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.249127 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.249092    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:46.624645 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:46.624618    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"]
Oct 30 21:19:46.626813 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:19:46.626780    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d81852_0839_460f_b0f7_0f8998dcc2da.slice/crio-b0541652b4027d350f9274663f1f938e2ea7dcf97e1ba8a90cafc44261c1dd19 WatchSource:0}: Error finding container b0541652b4027d350f9274663f1f938e2ea7dcf97e1ba8a90cafc44261c1dd19: Status 404 returned error can't find the container with id b0541652b4027d350f9274663f1f938e2ea7dcf97e1ba8a90cafc44261c1dd19
Oct 30 21:19:47.254371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:47.254275    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk" event={"ID":"00d81852-0839-460f-b0f7-0f8998dcc2da","Type":"ContainerStarted","Data":"081ac58bd44ad6451fd24c816de062c67df06fd09a341cc5c485e937ba958afd"}
Oct 30 21:19:47.254371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:47.254331    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk" event={"ID":"00d81852-0839-460f-b0f7-0f8998dcc2da","Type":"ContainerStarted","Data":"b0541652b4027d350f9274663f1f938e2ea7dcf97e1ba8a90cafc44261c1dd19"}
Oct 30 21:19:47.369997 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:47.369947    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk" podStartSLOduration=2.369933236 podStartE2EDuration="2.369933236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:47.368417264 +0000 UTC m=+3567.969219225" watchObservedRunningTime="2026-10-30 21:19:47.369933236 +0000 UTC m=+3567.970735201"
Oct 30 21:19:47.542372 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:47.542287    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026dc109-9b62-4048-8a3c-4ff60670d266" path="/var/lib/kubelet/pods/026dc109-9b62-4048-8a3c-4ff60670d266/volumes"
Oct 30 21:19:48.261159 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:48.261126    5967 generic.go:356] "Generic (PLEG): container finished" podID="00d81852-0839-460f-b0f7-0f8998dcc2da" containerID="081ac58bd44ad6451fd24c816de062c67df06fd09a341cc5c485e937ba958afd" exitCode=0
Oct 30 21:19:48.261638 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:48.261214    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk" event={"ID":"00d81852-0839-460f-b0f7-0f8998dcc2da","Type":"ContainerDied","Data":"081ac58bd44ad6451fd24c816de062c67df06fd09a341cc5c485e937ba958afd"}
Oct 30 21:19:49.268882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.268850    5967 generic.go:356] "Generic (PLEG): container finished" podID="268f598a-6473-4f25-9b10-041d68adb13c" containerID="4bc9cc6de179ff4f51005a9147c309f17ad590adba53818c8f3f8713bd5b18d6" exitCode=0
Oct 30 21:19:49.269292 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.268928    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" event={"ID":"268f598a-6473-4f25-9b10-041d68adb13c","Type":"ContainerDied","Data":"4bc9cc6de179ff4f51005a9147c309f17ad590adba53818c8f3f8713bd5b18d6"}
Oct 30 21:19:49.269292 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.268995    5967 scope.go:122] "RemoveContainer" containerID="58affc23eefae4d079780e1a24761e2c7f3c2bac18973679366840683cc5571d"
Oct 30 21:19:49.410682 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.410617    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:19:49.442134 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.442111    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:49.486903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.486873    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.486903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.486910    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92\" (UniqueName: \"kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92\") pod \"00d81852-0839-460f-b0f7-0f8998dcc2da\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") "
Oct 30 21:19:49.487162 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.486938    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca\") pod \"00d81852-0839-460f-b0f7-0f8998dcc2da\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") "
Oct 30 21:19:49.487162 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487133    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.487276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487221    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.487335 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487287    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.487389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487350    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.487443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487424    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca\") pod \"00d81852-0839-460f-b0f7-0f8998dcc2da\" (UID: \"00d81852-0839-460f-b0f7-0f8998dcc2da\") "
Oct 30 21:19:49.487504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487461    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.487565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487470    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.487626 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487574    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.487626 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487611    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.488109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487766    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.488109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487783    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.488109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.487829    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss\") pod \"268f598a-6473-4f25-9b10-041d68adb13c\" (UID: \"268f598a-6473-4f25-9b10-041d68adb13c\") "
Oct 30 21:19:49.488109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488008    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:49.488443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488114    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca" pod "00d81852-0839-460f-b0f7-0f8998dcc2da" (UID: "00d81852-0839-460f-b0f7-0f8998dcc2da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:49.488443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488147    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.488443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488414    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.488443 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488433    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.488664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488448    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.488664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488464    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268f598a-6473-4f25-9b10-041d68adb13c-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.488664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488476    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.488664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.488489    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.489552 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.489528    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca" pod "00d81852-0839-460f-b0f7-0f8998dcc2da" (UID: "00d81852-0839-460f-b0f7-0f8998dcc2da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:49.489804 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.489777    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92" pod "00d81852-0839-460f-b0f7-0f8998dcc2da" (UID: "00d81852-0839-460f-b0f7-0f8998dcc2da"). InnerVolumeSpecName "kube-api-access-s5x92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:49.489934 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.489904    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:49.490444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.490418    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "kube-api-access-dtkss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:49.490444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.490429    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:19:49.490572 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.490458    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy" pod "268f598a-6473-4f25-9b10-041d68adb13c" (UID: "268f598a-6473-4f25-9b10-041d68adb13c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.589813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.589776    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268f598a-6473-4f25-9b10-041d68adb13c-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.589813 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.589810    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268f598a-6473-4f25-9b10-041d68adb13c-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.590020 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.589826    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkss\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-kube-api-access-dtkss\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.590020 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.589840    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268f598a-6473-4f25-9b10-041d68adb13c-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.590020 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.589854    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5x92\" (UniqueName: \"kubernetes.io/projected/00d81852-0839-460f-b0f7-0f8998dcc2da-kube-api-access-s5x92\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.590020 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:49.589869    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d81852-0839-460f-b0f7-0f8998dcc2da-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:50.276256 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.276225    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"
Oct 30 21:19:50.276768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.276258    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk" event={"ID":"00d81852-0839-460f-b0f7-0f8998dcc2da","Type":"ContainerDied","Data":"b0541652b4027d350f9274663f1f938e2ea7dcf97e1ba8a90cafc44261c1dd19"}
Oct 30 21:19:50.276768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.276349    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0541652b4027d350f9274663f1f938e2ea7dcf97e1ba8a90cafc44261c1dd19"
Oct 30 21:19:50.277766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.277714    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt" event={"ID":"268f598a-6473-4f25-9b10-041d68adb13c","Type":"ContainerDied","Data":"4e316c9d5a591d479d3f594d30c44e62bddc23f9fdbaed2a11d913f41d2b9862"}
Oct 30 21:19:50.277910 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.277791    5967 scope.go:122] "RemoveContainer" containerID="4bc9cc6de179ff4f51005a9147c309f17ad590adba53818c8f3f8713bd5b18d6"
Oct 30 21:19:50.277910 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.277817    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"
Oct 30 21:19:50.542538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.542505    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"]
Oct 30 21:19:50.547802 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:50.547776    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt"]
Oct 30 21:19:51.541563 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:51.541531    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268f598a-6473-4f25-9b10-041d68adb13c" path="/var/lib/kubelet/pods/268f598a-6473-4f25-9b10-041d68adb13c/volumes"
Oct 30 21:19:53.054762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:53.054706    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"]
Oct 30 21:19:53.104125 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:53.104091    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dtjkg"]
Oct 30 21:19:53.542551 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:19:53.542507    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0" path="/var/lib/kubelet/pods/a8b61bad-ee86-4bd8-83ab-b7f8678c2ff0/volumes"
Oct 30 21:20:01.979346 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:20:01.979303    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:20:23.017115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:20:23.017075    5967 scope.go:122] "RemoveContainer" containerID="38651d74d975f7609e6fc290056bf9376e5368a46cfafa874dc9aea2110ae1f8"
Oct 30 21:20:23.026129 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:20:23.026108    5967 scope.go:122] "RemoveContainer" containerID="8ba356351b94155a08d2b7eac0182be43f5dc8abbddc8e6b8f4793342cda961c"
Oct 30 21:20:31.980801 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:20:31.980697    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:21:01.979966 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:21:01.979927    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:21:23.061364 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:23.061330    5967 scope.go:122] "RemoveContainer" containerID="c3475dcb44aa3cf43e2fb4b7f0195c46ac600c6390b451c6aa385242c47d6f4b"
Oct 30 21:21:31.980649 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:21:31.980608    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:21:32.769612 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.769580    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"]
Oct 30 21:21:32.770055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770042    5967 cpu_manager.go:414] "containerMap: removing container" podUID="00d81852-0839-460f-b0f7-0f8998dcc2da" containerName="revoke-keys"
Oct 30 21:21:32.770115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770056    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d81852-0839-460f-b0f7-0f8998dcc2da" containerName="revoke-keys"
Oct 30 21:21:32.770115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770071    5967 cpu_manager.go:414] "containerMap: removing container" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy"
Oct 30 21:21:32.770115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770076    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy"
Oct 30 21:21:32.770115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770091    5967 cpu_manager.go:414] "containerMap: removing container" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy"
Oct 30 21:21:32.770115 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770095    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy"
Oct 30 21:21:32.770329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770174    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d81852-0839-460f-b0f7-0f8998dcc2da" containerName="revoke-keys"
Oct 30 21:21:32.770329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770187    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy"
Oct 30 21:21:32.770329 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.770194    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="268f598a-6473-4f25-9b10-041d68adb13c" containerName="istio-proxy"
Oct 30 21:21:32.774462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.774447    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.826625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.826596    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:21:32.867689 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.867652    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.867689 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.867690    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8xt\" (UniqueName: \"kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.867942 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.867715    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.876143 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.876115    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:21:32.876290 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.876115    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:21:32.876290 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.876159    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:21:32.877983 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.877964    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"]
Oct 30 21:21:32.969353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.969322    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.969353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.969355    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8xt\" (UniqueName: \"kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.969594 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.969376    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.970019 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.969987    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:32.971866 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:32.971848    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:33.027815 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:33.027733    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8xt\" (UniqueName: \"kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt\") pod \"maas-api-revoke-sub-d576e68b-k7gq4\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:33.086958 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:33.086928    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:33.461185 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:33.461151    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"]
Oct 30 21:21:33.463136 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:21:33.463113    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb9dfc6_c513_4b54_afa3_7f0ac40bc039.slice/crio-170de25713d5c5333153f2e1fe5769b131addb0bd773be97523a68f6ffd74516 WatchSource:0}: Error finding container 170de25713d5c5333153f2e1fe5769b131addb0bd773be97523a68f6ffd74516: Status 404 returned error can't find the container with id 170de25713d5c5333153f2e1fe5769b131addb0bd773be97523a68f6ffd74516
Oct 30 21:21:33.872513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:33.872448    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4" event={"ID":"edb9dfc6-c513-4b54-afa3-7f0ac40bc039","Type":"ContainerStarted","Data":"dedc25cbaf29a4839a29e68e37f967b2343531cc8ea4f1662c8f529fdf155470"}
Oct 30 21:21:33.872843 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:33.872813    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4" event={"ID":"edb9dfc6-c513-4b54-afa3-7f0ac40bc039","Type":"ContainerStarted","Data":"170de25713d5c5333153f2e1fe5769b131addb0bd773be97523a68f6ffd74516"}
Oct 30 21:21:33.989371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:33.989316    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4" podStartSLOduration=1.989302263 podStartE2EDuration="1.989302263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:33.988139691 +0000 UTC m=+3674.588941651" watchObservedRunningTime="2026-10-30 21:21:33.989302263 +0000 UTC m=+3674.590104223"
Oct 30 21:21:34.879812 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:34.879774    5967 generic.go:356] "Generic (PLEG): container finished" podID="edb9dfc6-c513-4b54-afa3-7f0ac40bc039" containerID="dedc25cbaf29a4839a29e68e37f967b2343531cc8ea4f1662c8f529fdf155470" exitCode=0
Oct 30 21:21:34.880212 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:34.879821    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4" event={"ID":"edb9dfc6-c513-4b54-afa3-7f0ac40bc039","Type":"ContainerDied","Data":"dedc25cbaf29a4839a29e68e37f967b2343531cc8ea4f1662c8f529fdf155470"}
Oct 30 21:21:36.019128 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.019105    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:36.102853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.102808    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca\") pod \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") "
Oct 30 21:21:36.103045 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.102912    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt\" (UniqueName: \"kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt\") pod \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") "
Oct 30 21:21:36.103045 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.103000    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca\") pod \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\" (UID: \"edb9dfc6-c513-4b54-afa3-7f0ac40bc039\") "
Oct 30 21:21:36.103175 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.103150    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca" pod "edb9dfc6-c513-4b54-afa3-7f0ac40bc039" (UID: "edb9dfc6-c513-4b54-afa3-7f0ac40bc039"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:36.103352 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.103332    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:21:36.105165 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.105142    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca" pod "edb9dfc6-c513-4b54-afa3-7f0ac40bc039" (UID: "edb9dfc6-c513-4b54-afa3-7f0ac40bc039"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:36.105268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.105201    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt" pod "edb9dfc6-c513-4b54-afa3-7f0ac40bc039" (UID: "edb9dfc6-c513-4b54-afa3-7f0ac40bc039"). InnerVolumeSpecName "kube-api-access-zs8xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:36.204523 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.204442    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8xt\" (UniqueName: \"kubernetes.io/projected/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-kube-api-access-zs8xt\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:21:36.204523 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.204471    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb9dfc6-c513-4b54-afa3-7f0ac40bc039-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:21:36.891536 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.891504    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"
Oct 30 21:21:36.891536 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.891522    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4" event={"ID":"edb9dfc6-c513-4b54-afa3-7f0ac40bc039","Type":"ContainerDied","Data":"170de25713d5c5333153f2e1fe5769b131addb0bd773be97523a68f6ffd74516"}
Oct 30 21:21:36.891775 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:36.891575    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170de25713d5c5333153f2e1fe5769b131addb0bd773be97523a68f6ffd74516"
Oct 30 21:21:47.047094 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:47.047058    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"]
Oct 30 21:21:47.096597 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:47.096564    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9jxq8"]
Oct 30 21:21:47.541928 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:47.541897    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033000d1-1f7f-4038-89c8-70b9f44866e8" path="/var/lib/kubelet/pods/033000d1-1f7f-4038-89c8-70b9f44866e8/volumes"
Oct 30 21:21:58.046195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:58.046158    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"]
Oct 30 21:21:58.095667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:58.095632    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jrzxz"]
Oct 30 21:21:59.542962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:21:59.542889    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee7429b-b2e2-4607-8d7f-da1da49d9fc3" path="/var/lib/kubelet/pods/6ee7429b-b2e2-4607-8d7f-da1da49d9fc3/volumes"
Oct 30 21:22:01.980912 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:22:01.980876    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:22:02.007648 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.007613    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"]
Oct 30 21:22:02.008101 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.008087    5967 cpu_manager.go:414] "containerMap: removing container" podUID="edb9dfc6-c513-4b54-afa3-7f0ac40bc039" containerName="revoke-keys"
Oct 30 21:22:02.008187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.008102    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb9dfc6-c513-4b54-afa3-7f0ac40bc039" containerName="revoke-keys"
Oct 30 21:22:02.008255 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.008203    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb9dfc6-c513-4b54-afa3-7f0ac40bc039" containerName="revoke-keys"
Oct 30 21:22:02.011720 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.011701    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.063940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.063909    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:02.113434 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.113395    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:02.113625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.113445    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:02.113625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.113514    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:22:02.115273 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.115248    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"]
Oct 30 21:22:02.120722 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.120701    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.120909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.120731    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.120992 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.120968    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4ww\" (UniqueName: \"kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.222172 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.222136    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.222172 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.222172    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.222455 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.222231    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4ww\" (UniqueName: \"kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.222923 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.222902    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.224618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.224596    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.280379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.280306    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4ww\" (UniqueName: \"kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww\") pod \"maas-api-revoke-sub-d2bc93bc-qbv9l\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.324833 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.324792    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:02.701181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:02.701157    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"]
Oct 30 21:22:02.703623 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:22:02.703585    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8d8c97_135c_49a7_b0ac_4fe281e78882.slice/crio-f0ffd65c9ae7e3e3769157980dbf7eb178feb38a49d1d5cc0808bc8c1c6203ed WatchSource:0}: Error finding container f0ffd65c9ae7e3e3769157980dbf7eb178feb38a49d1d5cc0808bc8c1c6203ed: Status 404 returned error can't find the container with id f0ffd65c9ae7e3e3769157980dbf7eb178feb38a49d1d5cc0808bc8c1c6203ed
Oct 30 21:22:03.050872 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:03.050839    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l" event={"ID":"7e8d8c97-135c-49a7-b0ac-4fe281e78882","Type":"ContainerStarted","Data":"a3a12fdc266c994dbe5a10be750b461f9d934d702744ccbba33347c0ca61d0bd"}
Oct 30 21:22:03.051264 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:03.050901    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l" event={"ID":"7e8d8c97-135c-49a7-b0ac-4fe281e78882","Type":"ContainerStarted","Data":"f0ffd65c9ae7e3e3769157980dbf7eb178feb38a49d1d5cc0808bc8c1c6203ed"}
Oct 30 21:22:03.167445 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:03.167389    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l" podStartSLOduration=2.167371066 podStartE2EDuration="2.167371066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:03.167126728 +0000 UTC m=+3703.767928689" watchObservedRunningTime="2026-10-30 21:22:03.167371066 +0000 UTC m=+3703.768173028"
Oct 30 21:22:04.057856 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:04.057825    5967 generic.go:356] "Generic (PLEG): container finished" podID="7e8d8c97-135c-49a7-b0ac-4fe281e78882" containerID="a3a12fdc266c994dbe5a10be750b461f9d934d702744ccbba33347c0ca61d0bd" exitCode=0
Oct 30 21:22:04.058299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:04.057914    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l" event={"ID":"7e8d8c97-135c-49a7-b0ac-4fe281e78882","Type":"ContainerDied","Data":"a3a12fdc266c994dbe5a10be750b461f9d934d702744ccbba33347c0ca61d0bd"}
Oct 30 21:22:05.196341 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.196311    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:05.253590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.253559    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww\" (UniqueName: \"kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww\") pod \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") "
Oct 30 21:22:05.253590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.253592    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca\") pod \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") "
Oct 30 21:22:05.253836 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.253804    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca\") pod \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\" (UID: \"7e8d8c97-135c-49a7-b0ac-4fe281e78882\") "
Oct 30 21:22:05.254013 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.253990    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca" pod "7e8d8c97-135c-49a7-b0ac-4fe281e78882" (UID: "7e8d8c97-135c-49a7-b0ac-4fe281e78882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:05.254202 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.254185    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:05.255954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.255932    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww" pod "7e8d8c97-135c-49a7-b0ac-4fe281e78882" (UID: "7e8d8c97-135c-49a7-b0ac-4fe281e78882"). InnerVolumeSpecName "kube-api-access-fz4ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:05.256073 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.256007    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca" pod "7e8d8c97-135c-49a7-b0ac-4fe281e78882" (UID: "7e8d8c97-135c-49a7-b0ac-4fe281e78882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:05.355392 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.355306    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8d8c97-135c-49a7-b0ac-4fe281e78882-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:05.355392 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:05.355338    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4ww\" (UniqueName: \"kubernetes.io/projected/7e8d8c97-135c-49a7-b0ac-4fe281e78882-kube-api-access-fz4ww\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:06.069634 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:06.069602    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"
Oct 30 21:22:06.069864 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:06.069640    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l" event={"ID":"7e8d8c97-135c-49a7-b0ac-4fe281e78882","Type":"ContainerDied","Data":"f0ffd65c9ae7e3e3769157980dbf7eb178feb38a49d1d5cc0808bc8c1c6203ed"}
Oct 30 21:22:06.069864 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:06.069688    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ffd65c9ae7e3e3769157980dbf7eb178feb38a49d1d5cc0808bc8c1c6203ed"
Oct 30 21:22:15.230938 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.230904    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"]
Oct 30 21:22:15.231379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.231366    5967 cpu_manager.go:414] "containerMap: removing container" podUID="7e8d8c97-135c-49a7-b0ac-4fe281e78882" containerName="revoke-keys"
Oct 30 21:22:15.231424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.231379    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8d8c97-135c-49a7-b0ac-4fe281e78882" containerName="revoke-keys"
Oct 30 21:22:15.231489 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.231479    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8d8c97-135c-49a7-b0ac-4fe281e78882" containerName="revoke-keys"
Oct 30 21:22:15.234515 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.234498    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.286621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.286586    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-87bb59-openshift-default-dockercfg-cs884\""
Oct 30 21:22:15.339181 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.339134    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"]
Oct 30 21:22:15.361498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361466    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361505    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361535    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpg9\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361610    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361671    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361732 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361722    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361784    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.361929 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361820    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.362033 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.361956    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.463494 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463452    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.463706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463509    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.463706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463567    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.463706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463592    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.463706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463622    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpg9\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.463706 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463650    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463708    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463791    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463846    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.463980    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.464039    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.464104    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.464266    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.464552 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.464531    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.466088 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.466065    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.466183 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.466125    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.521610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.521528    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpg9\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.521610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.521585    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token\") pod \"e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") " pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.548291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.548260    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:15.985988 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.985957    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"]
Oct 30 21:22:15.987881 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:22:15.987854    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039f2fb0_e5a0_4614_99aa_4af46ada1c3c.slice/crio-34d16ea20b4a6bc2f29e41404a8d7b3938f721c5824397f4f9bf21370cf2cc83 WatchSource:0}: Error finding container 34d16ea20b4a6bc2f29e41404a8d7b3938f721c5824397f4f9bf21370cf2cc83: Status 404 returned error can't find the container with id 34d16ea20b4a6bc2f29e41404a8d7b3938f721c5824397f4f9bf21370cf2cc83
Oct 30 21:22:15.990089 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.990056    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:15.990195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.990135    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:15.990195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:15.990180    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:16.124402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:16.124363    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" event={"ID":"039f2fb0-e5a0-4614-99aa-4af46ada1c3c","Type":"ContainerStarted","Data":"6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"}
Oct 30 21:22:16.124571 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:16.124430    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" event={"ID":"039f2fb0-e5a0-4614-99aa-4af46ada1c3c","Type":"ContainerStarted","Data":"34d16ea20b4a6bc2f29e41404a8d7b3938f721c5824397f4f9bf21370cf2cc83"}
Oct 30 21:22:16.244494 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:16.244395    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podStartSLOduration=1.244378362 podStartE2EDuration="1.244378362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:16.242186119 +0000 UTC m=+3716.842988090" watchObservedRunningTime="2026-10-30 21:22:16.244378362 +0000 UTC m=+3716.845180389"
Oct 30 21:22:16.548936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:16.548844    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:17.554310 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:17.554274    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:18.137406 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:18.137377    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:18.138259 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:18.138244    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:23.082955 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:23.082924    5967 scope.go:122] "RemoveContainer" containerID="90dc6bf68f70991dcff24cea37c3a5f55033047ee4da4824719c91014da56564"
Oct 30 21:22:23.092321 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:23.092302    5967 scope.go:122] "RemoveContainer" containerID="db1d4f7320326b300d7d16b0c7ccedee8a3e1ee849fe77427d9d57d690e18609"
Oct 30 21:22:31.979729 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:22:31.979693    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:22:43.274480 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:43.274449    5967 generic.go:356] "Generic (PLEG): container finished" podID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerID="6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c" exitCode=0
Oct 30 21:22:43.274950 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:43.274534    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" event={"ID":"039f2fb0-e5a0-4614-99aa-4af46ada1c3c","Type":"ContainerDied","Data":"6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"}
Oct 30 21:22:43.274996 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:43.274970    5967 scope.go:122] "RemoveContainer" containerID="6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"
Oct 30 21:22:43.276172 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:43.276145    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:43.276238 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:43.276214    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:43.276279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:43.276247    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:44.300513 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:44.300480    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" event={"ID":"039f2fb0-e5a0-4614-99aa-4af46ada1c3c","Type":"ContainerStarted","Data":"362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669"}
Oct 30 21:22:44.548861 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:44.548823    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:44.549964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:44.549946    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:22:44.550039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:44.549987    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:22:45.549197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:45.549161    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:22:45.549631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:45.549229    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:22:46.548719 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:46.548680    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:22:46.548936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:46.548753    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:22:47.387768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.385195    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"]
Oct 30 21:22:47.395381 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.395350    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.434552 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.434517    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.434889 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.434830    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtpg\" (UniqueName: \"kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.435123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.435072    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.435353 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.435247    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.435784 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.435488    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.435957 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.435941    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.439319 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.437955    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"]
Oct 30 21:22:47.494767 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.494489    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-87bb59\"/\"default-dockercfg-lssxg\""
Oct 30 21:22:47.494767 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.494683    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-87bb59\"/\"simulator-single-main-87bb59-kserve-self-signed-certs\""
Oct 30 21:22:47.499311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.498977    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-87bb59\"/\"kube-root-ca.crt\""
Oct 30 21:22:47.499311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.499174    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-87bb59\"/\"openshift-service-ca.crt\""
Oct 30 21:22:47.536682 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.536642    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.536933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.536693    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtpg\" (UniqueName: \"kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.536933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.536783    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.536933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.536825    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.536933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.536867    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.536933 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.536916    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.537915 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.537887    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.538040 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.537934    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.538109 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.538071    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.540774 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.540207    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.542618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.542600    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.548836 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.548622    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:22:47.548836 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.548702    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:22:47.600195 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.600159    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtpg\" (UniqueName: \"kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg\") pod \"simulator-single-main-87bb59-kserve-684c7845f7-xs9tt\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") " pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:47.709419 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:47.709329    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:48.099494 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.098812    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"]
Oct 30 21:22:48.101451 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:22:48.101417    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d743f2_0032_404c_9834_6fcfa50a496c.slice/crio-0d3185ae385b5d2e6a410bab2b51aecc36ae7ed8770e1d0798fb594ab8ec12a1 WatchSource:0}: Error finding container 0d3185ae385b5d2e6a410bab2b51aecc36ae7ed8770e1d0798fb594ab8ec12a1: Status 404 returned error can't find the container with id 0d3185ae385b5d2e6a410bab2b51aecc36ae7ed8770e1d0798fb594ab8ec12a1
Oct 30 21:22:48.137546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.137517    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:48.325456 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.325414    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" event={"ID":"a4d743f2-0032-404c-9834-6fcfa50a496c","Type":"ContainerStarted","Data":"0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665"}
Oct 30 21:22:48.325650 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.325468    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" event={"ID":"a4d743f2-0032-404c-9834-6fcfa50a496c","Type":"ContainerStarted","Data":"0d3185ae385b5d2e6a410bab2b51aecc36ae7ed8770e1d0798fb594ab8ec12a1"}
Oct 30 21:22:48.442605 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.442500    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" podStartSLOduration=1.442484112 podStartE2EDuration="1.442484112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:48.441036089 +0000 UTC m=+3749.041838048" watchObservedRunningTime="2026-10-30 21:22:48.442484112 +0000 UTC m=+3749.043286072"
Oct 30 21:22:48.548838 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.548808    5967 patch_prober.go:29] interesting pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:22:48.549005 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.548863    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:22:48.710470 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:48.710385    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:49.553030 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:49.552995    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:49.554046 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:49.554026    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:22:57.710451 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:57.710368    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:57.723215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:57.723188    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:22:58.392589 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:22:58.392558    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:23:01.980489 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:23:01.980448    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:23:23.121132 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:23.121099    5967 scope.go:122] "RemoveContainer" containerID="9d836b50395e25b560f85707dcc61fa87da071cd78ca576c35a4f0c8b33b5e44"
Oct 30 21:23:23.133620 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:23.133600    5967 scope.go:122] "RemoveContainer" containerID="a2fe207c1f3e33386ac3f9ebdf8f4a61b730d9c81172db3f8bc08b2d3fd2f9a0"
Oct 30 21:23:23.141905 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:23.141886    5967 scope.go:122] "RemoveContainer" containerID="333a7ef395d67c2d3316ed9f967d25233c00d23a901e0a92144d6582dc1580d4"
Oct 30 21:23:31.979644 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:23:31.979566    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:23:34.236064 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.236036    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"]
Oct 30 21:23:34.241322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.241303    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.315186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.315153    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptnq\" (UniqueName: \"kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.315349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.315215    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.315427 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.315371    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.344154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.344127    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:34.344154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.344126    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:34.344154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.344150    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:23:34.344336 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.344126    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:34.346012 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.345982    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"]
Oct 30 21:23:34.416341 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.416310    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptnq\" (UniqueName: \"kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.416493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.416388    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.416493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.416482    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.417010 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.416993    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.418936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.418921    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.472827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.472802    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptnq\" (UniqueName: \"kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq\") pod \"maas-api-revoke-sub-aac991e0-tm9tt\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.554313 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.554260    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:34.931720 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:34.931697    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"]
Oct 30 21:23:34.933651 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:23:34.933619    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024a8d09_1293_44ca_839d_3ccd0573cada.slice/crio-053dbc436829eb9a22a77179b2582e26866cae386b8627ece21ff46b31c77ff1 WatchSource:0}: Error finding container 053dbc436829eb9a22a77179b2582e26866cae386b8627ece21ff46b31c77ff1: Status 404 returned error can't find the container with id 053dbc436829eb9a22a77179b2582e26866cae386b8627ece21ff46b31c77ff1
Oct 30 21:23:35.583428 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:35.583399    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt" event={"ID":"024a8d09-1293-44ca-839d-3ccd0573cada","Type":"ContainerStarted","Data":"319e6cc615ce497c47a428377a9ec9e881205a0e695ae6c23f6f17134190a49c"}
Oct 30 21:23:35.583825 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:35.583446    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt" event={"ID":"024a8d09-1293-44ca-839d-3ccd0573cada","Type":"ContainerStarted","Data":"053dbc436829eb9a22a77179b2582e26866cae386b8627ece21ff46b31c77ff1"}
Oct 30 21:23:35.701769 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:35.701705    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt" podStartSLOduration=1.701691593 podStartE2EDuration="1.701691593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:35.699970671 +0000 UTC m=+3796.300772632" watchObservedRunningTime="2026-10-30 21:23:35.701691593 +0000 UTC m=+3796.302493946"
Oct 30 21:23:36.597943 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:36.597911    5967 generic.go:356] "Generic (PLEG): container finished" podID="024a8d09-1293-44ca-839d-3ccd0573cada" containerID="319e6cc615ce497c47a428377a9ec9e881205a0e695ae6c23f6f17134190a49c" exitCode=0
Oct 30 21:23:36.598345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:36.597992    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt" event={"ID":"024a8d09-1293-44ca-839d-3ccd0573cada","Type":"ContainerDied","Data":"319e6cc615ce497c47a428377a9ec9e881205a0e695ae6c23f6f17134190a49c"}
Oct 30 21:23:37.743502 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.743476    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:37.847503 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.847481    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca\") pod \"024a8d09-1293-44ca-839d-3ccd0573cada\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") "
Oct 30 21:23:37.847666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.847552    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq\" (UniqueName: \"kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq\") pod \"024a8d09-1293-44ca-839d-3ccd0573cada\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") "
Oct 30 21:23:37.847666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.847599    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca\") pod \"024a8d09-1293-44ca-839d-3ccd0573cada\" (UID: \"024a8d09-1293-44ca-839d-3ccd0573cada\") "
Oct 30 21:23:37.847969 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.847947    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca" pod "024a8d09-1293-44ca-839d-3ccd0573cada" (UID: "024a8d09-1293-44ca-839d-3ccd0573cada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:37.849687 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.849665    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca" pod "024a8d09-1293-44ca-839d-3ccd0573cada" (UID: "024a8d09-1293-44ca-839d-3ccd0573cada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:37.849786 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.849722    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq" pod "024a8d09-1293-44ca-839d-3ccd0573cada" (UID: "024a8d09-1293-44ca-839d-3ccd0573cada"). InnerVolumeSpecName "kube-api-access-2ptnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:37.949261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.949237    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.949261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.949261    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024a8d09-1293-44ca-839d-3ccd0573cada-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.949440 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:37.949271    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptnq\" (UniqueName: \"kubernetes.io/projected/024a8d09-1293-44ca-839d-3ccd0573cada-kube-api-access-2ptnq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:38.608713 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:38.608676    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt" event={"ID":"024a8d09-1293-44ca-839d-3ccd0573cada","Type":"ContainerDied","Data":"053dbc436829eb9a22a77179b2582e26866cae386b8627ece21ff46b31c77ff1"}
Oct 30 21:23:38.608900 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:38.608729    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="053dbc436829eb9a22a77179b2582e26866cae386b8627ece21ff46b31c77ff1"
Oct 30 21:23:38.608900 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:38.608689    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"
Oct 30 21:23:45.944291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:45.944251    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"]
Oct 30 21:23:45.944947 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:45.944927    5967 cpu_manager.go:414] "containerMap: removing container" podUID="024a8d09-1293-44ca-839d-3ccd0573cada" containerName="revoke-keys"
Oct 30 21:23:45.945070 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:45.944948    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024a8d09-1293-44ca-839d-3ccd0573cada" containerName="revoke-keys"
Oct 30 21:23:45.945132 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:45.945107    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="024a8d09-1293-44ca-839d-3ccd0573cada" containerName="revoke-keys"
Oct 30 21:23:45.948450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:45.948428    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.000600 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.000570    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:46.028191 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.028161    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.028349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.028224    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.028349 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.028310    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnwh\" (UniqueName: \"kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.050270 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.050243    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:46.050440 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.050309    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:46.050440 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.050320    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:23:46.052418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.052397    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"]
Oct 30 21:23:46.129727 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.129698    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.129902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.129779    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnwh\" (UniqueName: \"kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.129946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.129906    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.130610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.130546    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.132430 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.132405    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.186397 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.186375    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnwh\" (UniqueName: \"kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.261610 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.261587    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:46.634769 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.634715    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"]
Oct 30 21:23:46.636600 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:23:46.636577    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18aeedf5_66ea_48d4_ba72_af1e36cd0400.slice/crio-b271fede3264aa70367a800753c8aa183ffe23f1eeb182f390a81c7d5f15a5c0 WatchSource:0}: Error finding container b271fede3264aa70367a800753c8aa183ffe23f1eeb182f390a81c7d5f15a5c0: Status 404 returned error can't find the container with id b271fede3264aa70367a800753c8aa183ffe23f1eeb182f390a81c7d5f15a5c0
Oct 30 21:23:46.652334 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:46.652314    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn" event={"ID":"18aeedf5-66ea-48d4-ba72-af1e36cd0400","Type":"ContainerStarted","Data":"b271fede3264aa70367a800753c8aa183ffe23f1eeb182f390a81c7d5f15a5c0"}
Oct 30 21:23:47.659252 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:47.659216    5967 generic.go:356] "Generic (PLEG): container finished" podID="18aeedf5-66ea-48d4-ba72-af1e36cd0400" containerID="c107ae9013b69ee68b861c3159510109060fae7eed3d8a37b5b4364e177e73b2" exitCode=0
Oct 30 21:23:47.659643 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:47.659258    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn" event={"ID":"18aeedf5-66ea-48d4-ba72-af1e36cd0400","Type":"ContainerDied","Data":"c107ae9013b69ee68b861c3159510109060fae7eed3d8a37b5b4364e177e73b2"}
Oct 30 21:23:48.805392 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.805369    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:48.960896 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.960821    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca\") pod \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") "
Oct 30 21:23:48.961043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.960948    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh\" (UniqueName: \"kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh\") pod \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") "
Oct 30 21:23:48.961043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.961013    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca\") pod \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\" (UID: \"18aeedf5-66ea-48d4-ba72-af1e36cd0400\") "
Oct 30 21:23:48.961227 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.961200    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca" pod "18aeedf5-66ea-48d4-ba72-af1e36cd0400" (UID: "18aeedf5-66ea-48d4-ba72-af1e36cd0400"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:48.961460 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.961443    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.963103 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.963082    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh" pod "18aeedf5-66ea-48d4-ba72-af1e36cd0400" (UID: "18aeedf5-66ea-48d4-ba72-af1e36cd0400"). InnerVolumeSpecName "kube-api-access-nbnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.963196 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:48.963114    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca" pod "18aeedf5-66ea-48d4-ba72-af1e36cd0400" (UID: "18aeedf5-66ea-48d4-ba72-af1e36cd0400"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:49.062936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:49.062908    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnwh\" (UniqueName: \"kubernetes.io/projected/18aeedf5-66ea-48d4-ba72-af1e36cd0400-kube-api-access-nbnwh\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.062936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:49.062933    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18aeedf5-66ea-48d4-ba72-af1e36cd0400-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.671673 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:49.671644    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"
Oct 30 21:23:49.671673 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:49.671670    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn" event={"ID":"18aeedf5-66ea-48d4-ba72-af1e36cd0400","Type":"ContainerDied","Data":"b271fede3264aa70367a800753c8aa183ffe23f1eeb182f390a81c7d5f15a5c0"}
Oct 30 21:23:49.671905 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:49.671709    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b271fede3264aa70367a800753c8aa183ffe23f1eeb182f390a81c7d5f15a5c0"
Oct 30 21:23:50.344197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:23:50.344160    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"]
Oct 30 21:24:01.981078 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:24:01.981037    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:24:05.158285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.158252    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"]
Oct 30 21:24:05.207222 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.207198    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-87bb59-szgkn"]
Oct 30 21:24:05.364616 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.364552    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" podUID="a4d743f2-0032-404c-9834-6fcfa50a496c" containerName="main" containerID="cri-o://0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665" gracePeriod=45
Oct 30 21:24:05.382084 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.382059    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" podUID="a4d743f2-0032-404c-9834-6fcfa50a496c" containerName="main" probeResult="failure" output="Get \"https://10.134.1.3:8000/ready\": dial tcp 10.134.1.3:8000: connect: connection refused"
Oct 30 21:24:05.544976 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.544944    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18aeedf5-66ea-48d4-ba72-af1e36cd0400" path="/var/lib/kubelet/pods/18aeedf5-66ea-48d4-ba72-af1e36cd0400/volumes"
Oct 30 21:24:05.609008 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.608986    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:24:05.719964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.719935    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir\") pod \"a4d743f2-0032-404c-9834-6fcfa50a496c\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") "
Oct 30 21:24:05.719964 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.719966    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm\") pod \"a4d743f2-0032-404c-9834-6fcfa50a496c\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") "
Oct 30 21:24:05.720156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.719987    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache\") pod \"a4d743f2-0032-404c-9834-6fcfa50a496c\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") "
Oct 30 21:24:05.720156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720024    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg\" (UniqueName: \"kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg\") pod \"a4d743f2-0032-404c-9834-6fcfa50a496c\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") "
Oct 30 21:24:05.720156 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720071    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home\") pod \"a4d743f2-0032-404c-9834-6fcfa50a496c\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") "
Oct 30 21:24:05.720324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720226    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs\") pod \"a4d743f2-0032-404c-9834-6fcfa50a496c\" (UID: \"a4d743f2-0032-404c-9834-6fcfa50a496c\") "
Oct 30 21:24:05.720324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720225    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache" pod "a4d743f2-0032-404c-9834-6fcfa50a496c" (UID: "a4d743f2-0032-404c-9834-6fcfa50a496c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:05.720324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720233    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir" pod "a4d743f2-0032-404c-9834-6fcfa50a496c" (UID: "a4d743f2-0032-404c-9834-6fcfa50a496c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:05.720486 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720405    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home" pod "a4d743f2-0032-404c-9834-6fcfa50a496c" (UID: "a4d743f2-0032-404c-9834-6fcfa50a496c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:05.720608 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720586    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.720840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720613    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.720840 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.720625    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.722322 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.722300    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm" pod "a4d743f2-0032-404c-9834-6fcfa50a496c" (UID: "a4d743f2-0032-404c-9834-6fcfa50a496c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:05.722439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.722336    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs" pod "a4d743f2-0032-404c-9834-6fcfa50a496c" (UID: "a4d743f2-0032-404c-9834-6fcfa50a496c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:05.722439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.722368    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg" pod "a4d743f2-0032-404c-9834-6fcfa50a496c" (UID: "a4d743f2-0032-404c-9834-6fcfa50a496c"). InnerVolumeSpecName "kube-api-access-fmtpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:05.756890 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.756858    5967 generic.go:356] "Generic (PLEG): container finished" podID="a4d743f2-0032-404c-9834-6fcfa50a496c" containerID="0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665" exitCode=0
Oct 30 21:24:05.757061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.756916    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"
Oct 30 21:24:05.757061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.756941    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" event={"ID":"a4d743f2-0032-404c-9834-6fcfa50a496c","Type":"ContainerDied","Data":"0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665"}
Oct 30 21:24:05.757061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.756981    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt" event={"ID":"a4d743f2-0032-404c-9834-6fcfa50a496c","Type":"ContainerDied","Data":"0d3185ae385b5d2e6a410bab2b51aecc36ae7ed8770e1d0798fb594ab8ec12a1"}
Oct 30 21:24:05.757061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.756999    5967 scope.go:122] "RemoveContainer" containerID="0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665"
Oct 30 21:24:05.767981 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.767000    5967 scope.go:122] "RemoveContainer" containerID="0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665"
Oct 30 21:24:05.767981 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:24:05.767292    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665\": container with ID starting with 0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665 not found: ID does not exist" containerID="0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665"
Oct 30 21:24:05.767981 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.767316    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665"} err="failed to get container status \"0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665\": rpc error: code = NotFound desc = could not find container \"0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665\": container with ID starting with 0c800df6de8f7ed26b8f6428992680037656fb10daa7b049fe2ecd4338093665 not found: ID does not exist"
Oct 30 21:24:05.821591 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.821566    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d743f2-0032-404c-9834-6fcfa50a496c-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.821591 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.821591    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmtpg\" (UniqueName: \"kubernetes.io/projected/a4d743f2-0032-404c-9834-6fcfa50a496c-kube-api-access-fmtpg\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.821591 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.821603    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d743f2-0032-404c-9834-6fcfa50a496c-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.918618 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.918592    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"]
Oct 30 21:24:05.922717 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:05.922696    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-87bb59/simulator-single-main-87bb59-kserve-684c7845f7-xs9tt"]
Oct 30 21:24:07.541689 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:07.541655    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d743f2-0032-404c-9834-6fcfa50a496c" path="/var/lib/kubelet/pods/a4d743f2-0032-404c-9834-6fcfa50a496c/volumes"
Oct 30 21:24:09.045956 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:09.045917    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"]
Oct 30 21:24:09.095368 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:09.095333    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47bda64f-c4pl7"]
Oct 30 21:24:09.541987 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:09.541946    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e92c6f8-d9fd-4055-a478-f4f646911a91" path="/var/lib/kubelet/pods/9e92c6f8-d9fd-4055-a478-f4f646911a91/volumes"
Oct 30 21:24:11.538244 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:11.538208    5967 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" secret="" err="secret \"e2e-worker-single-main-87bb59-openshift-default-dockercfg-cs884\" not found"
Oct 30 21:24:11.580102 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:11.580069    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"]
Oct 30 21:24:11.789535 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:11.789415    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy" containerID="cri-o://362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669" gracePeriod=30
Oct 30 21:24:17.041178 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.041146    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:24:17.134187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134109    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134187 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134171    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134202    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134248    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134271    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134293    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134333    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134392    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134425    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket\") pod \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\" (UID: \"039f2fb0-e5a0-4614-99aa-4af46ada1c3c\") "
Oct 30 21:24:17.134892 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134557    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.134892 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134580    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.134892 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134564    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.135233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134896    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.135233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134920    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.135233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.134935    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.135406 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.135309    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.135472 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.135427    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:17.136791 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.136760    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.136917 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.136793    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:17.136917 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.136831    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:17.138995 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.138974    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9" pod "039f2fb0-e5a0-4614-99aa-4af46ada1c3c" (UID: "039f2fb0-e5a0-4614-99aa-4af46ada1c3c"). InnerVolumeSpecName "kube-api-access-tmpg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:17.235701 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.235672    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.235701 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.235699    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.235701 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.235708    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.235951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.235718    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.235951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.235728    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.235951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.235765    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpg9\" (UniqueName: \"kubernetes.io/projected/039f2fb0-e5a0-4614-99aa-4af46ada1c3c-kube-api-access-tmpg9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.820959 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.820869    5967 generic.go:356] "Generic (PLEG): container finished" podID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerID="362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669" exitCode=0
Oct 30 21:24:17.820959 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.820924    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"
Oct 30 21:24:17.820959 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.820943    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" event={"ID":"039f2fb0-e5a0-4614-99aa-4af46ada1c3c","Type":"ContainerDied","Data":"362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669"}
Oct 30 21:24:17.821271 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.820998    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j" event={"ID":"039f2fb0-e5a0-4614-99aa-4af46ada1c3c","Type":"ContainerDied","Data":"34d16ea20b4a6bc2f29e41404a8d7b3938f721c5824397f4f9bf21370cf2cc83"}
Oct 30 21:24:17.821271 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.821018    5967 scope.go:122] "RemoveContainer" containerID="362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669"
Oct 30 21:24:17.830493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.830472    5967 scope.go:122] "RemoveContainer" containerID="6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"
Oct 30 21:24:17.838573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.838557    5967 scope.go:122] "RemoveContainer" containerID="362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669"
Oct 30 21:24:17.838829 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:24:17.838812    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669\": container with ID starting with 362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669 not found: ID does not exist" containerID="362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669"
Oct 30 21:24:17.838885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.838842    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669"} err="failed to get container status \"362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669\": rpc error: code = NotFound desc = could not find container \"362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669\": container with ID starting with 362594d2b7839b35107a1b4ba9af3836e570543b5d95b92ec903a3e035208669 not found: ID does not exist"
Oct 30 21:24:17.838885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.838859    5967 scope.go:122] "RemoveContainer" containerID="6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"
Oct 30 21:24:17.839063 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:24:17.839047    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c\": container with ID starting with 6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c not found: ID does not exist" containerID="6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"
Oct 30 21:24:17.839104 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:17.839066    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c"} err="failed to get container status \"6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c\": rpc error: code = NotFound desc = could not find container \"6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c\": container with ID starting with 6621cce20cf6cccae1efce7bd3ce8828c02e8368627b4b3ccb89013cc361cf4c not found: ID does not exist"
Oct 30 21:24:18.037096 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.037056    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"]
Oct 30 21:24:18.085192 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.085110    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j"]
Oct 30 21:24:18.868170 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868141    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"]
Oct 30 21:24:18.868602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868590    5967 cpu_manager.go:414] "containerMap: removing container" podUID="a4d743f2-0032-404c-9834-6fcfa50a496c" containerName="main"
Oct 30 21:24:18.868660 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868604    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d743f2-0032-404c-9834-6fcfa50a496c" containerName="main"
Oct 30 21:24:18.868660 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868617    5967 cpu_manager.go:414] "containerMap: removing container" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy"
Oct 30 21:24:18.868660 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868622    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy"
Oct 30 21:24:18.868660 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868644    5967 cpu_manager.go:414] "containerMap: removing container" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy"
Oct 30 21:24:18.868660 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868649    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy"
Oct 30 21:24:18.868857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868664    5967 cpu_manager.go:414] "containerMap: removing container" podUID="18aeedf5-66ea-48d4-ba72-af1e36cd0400" containerName="revoke-keys"
Oct 30 21:24:18.868857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868669    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18aeedf5-66ea-48d4-ba72-af1e36cd0400" containerName="revoke-keys"
Oct 30 21:24:18.868857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868753    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy"
Oct 30 21:24:18.868857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868766    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="18aeedf5-66ea-48d4-ba72-af1e36cd0400" containerName="revoke-keys"
Oct 30 21:24:18.868857 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.868774    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d743f2-0032-404c-9834-6fcfa50a496c" containerName="main"
Oct 30 21:24:18.873502 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.873484    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.953714 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.953685    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z724\" (UniqueName: \"kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.953879 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.953747    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.953879 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.953816    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.953879 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.953840    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.954060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.953880    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.954060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.953938    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:18.974844 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.974819    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-569edc7e-kserve-self-signed-certs\""
Oct 30 21:24:18.976677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:18.976658    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"]
Oct 30 21:24:19.054873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.054841    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z724\" (UniqueName: \"kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.054885    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.054907    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.054928    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.054958    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.054979    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.055283    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055371 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.055346    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.055483 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.055462    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.057365 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.057341    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.057505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.057489    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.112583 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.112554    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z724\" (UniqueName: \"kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724\") pod \"e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") " pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.184873 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.184810    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:19.542089 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.542059    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" path="/var/lib/kubelet/pods/039f2fb0-e5a0-4614-99aa-4af46ada1c3c/volumes"
Oct 30 21:24:19.563072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.563051    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"]
Oct 30 21:24:19.565150 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:24:19.565121    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c6f22a9_c6ae_4afb_9341_5165802b8b04.slice/crio-f13412dac0142c1508ec9e32a853f581388eee4fb1f20f3815411ea3f6775411 WatchSource:0}: Error finding container f13412dac0142c1508ec9e32a853f581388eee4fb1f20f3815411ea3f6775411: Status 404 returned error can't find the container with id f13412dac0142c1508ec9e32a853f581388eee4fb1f20f3815411ea3f6775411
Oct 30 21:24:19.835298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.835217    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" event={"ID":"9c6f22a9-c6ae-4afb-9341-5165802b8b04","Type":"ContainerStarted","Data":"b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8"}
Oct 30 21:24:19.835298 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.835259    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" event={"ID":"9c6f22a9-c6ae-4afb-9341-5165802b8b04","Type":"ContainerStarted","Data":"f13412dac0142c1508ec9e32a853f581388eee4fb1f20f3815411ea3f6775411"}
Oct 30 21:24:19.950800 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:19.950728    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" podStartSLOduration=1.950715396 podStartE2EDuration="1.950715396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:19.949398808 +0000 UTC m=+3840.550200769" watchObservedRunningTime="2026-10-30 21:24:19.950715396 +0000 UTC m=+3840.551517355"
Oct 30 21:24:20.185390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:20.185320    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:29.185872 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:29.185836    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:29.198300 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:29.198275    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:29.910442 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:29.910403    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:24:31.980231 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:24:31.980188    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:24:50.045700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:50.045666    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"]
Oct 30 21:24:50.094893 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:50.094866    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cnmpk"]
Oct 30 21:24:51.541393 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:24:51.541359    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d81852-0839-460f-b0f7-0f8998dcc2da" path="/var/lib/kubelet/pods/00d81852-0839-460f-b0f7-0f8998dcc2da/volumes"
Oct 30 21:25:01.979101 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:25:01.979024    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:25:23.192045 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:23.192015    5967 scope.go:122] "RemoveContainer" containerID="dadc3930c0b7098372f1df1d821aeb670c3d453868b61c1b1b32a0847e543342"
Oct 30 21:25:23.200841 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:23.200823    5967 scope.go:122] "RemoveContainer" containerID="4e9d4975fd32c856088dccb4e2e6b40b40ec78ab7d2508821b107d207dbd38d1"
Oct 30 21:25:31.982555 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:25:31.982509    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:25:35.666902 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.666869    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"]
Oct 30 21:25:35.667388 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.667374    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="039f2fb0-e5a0-4614-99aa-4af46ada1c3c" containerName="istio-proxy"
Oct 30 21:25:35.670351 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.670336    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.722273 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.722246    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:35.771729 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.771694    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:35.771729 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.771716    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:35.771936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.771786    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:25:35.773477 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.773457    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"]
Oct 30 21:25:35.830762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.830714    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmhq\" (UniqueName: \"kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.830920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.830807    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.830920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.830885    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.932268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.932200    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmhq\" (UniqueName: \"kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.932268 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.932246    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.932462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.932299    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.932937 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.932919    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.934842 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.934819    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:35.988692 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:35.988665    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmhq\" (UniqueName: \"kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq\") pod \"maas-api-revoke-sub-452b6d7b-wltdm\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:36.283278 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:36.283250    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:36.651110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:36.651087    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"]
Oct 30 21:25:36.653730 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:25:36.653702    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04172ef_0ae4_4fe1_9734_500abe12c0e8.slice/crio-5b3ffba388d7443b0838a91d6814c0a618d023b36510d4deeea1dab9a93d697a WatchSource:0}: Error finding container 5b3ffba388d7443b0838a91d6814c0a618d023b36510d4deeea1dab9a93d697a: Status 404 returned error can't find the container with id 5b3ffba388d7443b0838a91d6814c0a618d023b36510d4deeea1dab9a93d697a
Oct 30 21:25:37.296471 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:37.296384    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm" event={"ID":"f04172ef-0ae4-4fe1-9734-500abe12c0e8","Type":"ContainerStarted","Data":"dee43ce66a424f7d0ed1bf3465ccaf16a6e568084a9f900fefb217896bd393a4"}
Oct 30 21:25:37.296471 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:37.296451    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm" event={"ID":"f04172ef-0ae4-4fe1-9734-500abe12c0e8","Type":"ContainerStarted","Data":"5b3ffba388d7443b0838a91d6814c0a618d023b36510d4deeea1dab9a93d697a"}
Oct 30 21:25:37.412656 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:37.412599    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm" podStartSLOduration=2.412584309 podStartE2EDuration="2.412584309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:37.410794843 +0000 UTC m=+3918.011596806" watchObservedRunningTime="2026-10-30 21:25:37.412584309 +0000 UTC m=+3918.013386268"
Oct 30 21:25:38.338197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:38.338164    5967 generic.go:356] "Generic (PLEG): container finished" podID="f04172ef-0ae4-4fe1-9734-500abe12c0e8" containerID="dee43ce66a424f7d0ed1bf3465ccaf16a6e568084a9f900fefb217896bd393a4" exitCode=0
Oct 30 21:25:38.338819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:38.338793    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm" event={"ID":"f04172ef-0ae4-4fe1-9734-500abe12c0e8","Type":"ContainerDied","Data":"dee43ce66a424f7d0ed1bf3465ccaf16a6e568084a9f900fefb217896bd393a4"}
Oct 30 21:25:39.741876 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.741848    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:39.751010 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.750988    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca\") pod \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") "
Oct 30 21:25:39.751126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.751091    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq\" (UniqueName: \"kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq\") pod \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") "
Oct 30 21:25:39.751126 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.751112    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca\") pod \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\" (UID: \"f04172ef-0ae4-4fe1-9734-500abe12c0e8\") "
Oct 30 21:25:39.751517 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.751496    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca" pod "f04172ef-0ae4-4fe1-9734-500abe12c0e8" (UID: "f04172ef-0ae4-4fe1-9734-500abe12c0e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:39.753280 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.753260    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca" pod "f04172ef-0ae4-4fe1-9734-500abe12c0e8" (UID: "f04172ef-0ae4-4fe1-9734-500abe12c0e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:39.756067 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.756050    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq" pod "f04172ef-0ae4-4fe1-9734-500abe12c0e8" (UID: "f04172ef-0ae4-4fe1-9734-500abe12c0e8"). InnerVolumeSpecName "kube-api-access-fkmhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:39.852439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.852414    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmhq\" (UniqueName: \"kubernetes.io/projected/f04172ef-0ae4-4fe1-9734-500abe12c0e8-kube-api-access-fkmhq\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:39.852439 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.852443    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:39.852607 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:39.852452    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04172ef-0ae4-4fe1-9734-500abe12c0e8-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:40.352250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:40.352218    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"
Oct 30 21:25:40.352432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:40.352253    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm" event={"ID":"f04172ef-0ae4-4fe1-9734-500abe12c0e8","Type":"ContainerDied","Data":"5b3ffba388d7443b0838a91d6814c0a618d023b36510d4deeea1dab9a93d697a"}
Oct 30 21:25:40.352432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:40.352294    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b3ffba388d7443b0838a91d6814c0a618d023b36510d4deeea1dab9a93d697a"
Oct 30 21:25:48.631565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:48.631535    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"]
Oct 30 21:25:59.544733 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.544705    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"]
Oct 30 21:25:59.545283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.545249    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f04172ef-0ae4-4fe1-9734-500abe12c0e8" containerName="revoke-keys"
Oct 30 21:25:59.545283 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.545266    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04172ef-0ae4-4fe1-9734-500abe12c0e8" containerName="revoke-keys"
Oct 30 21:25:59.545401 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.545387    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04172ef-0ae4-4fe1-9734-500abe12c0e8" containerName="revoke-keys"
Oct 30 21:25:59.548619 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.548601    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.603412 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.603384    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:59.652962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.652931    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:59.652962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.652914    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:59.652962 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.652961    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:25:59.654771 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.654751    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"]
Oct 30 21:25:59.686091 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.686068    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmkh\" (UniqueName: \"kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.686212 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.686186    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.686260 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.686217    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.787145 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.787120    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmkh\" (UniqueName: \"kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.787258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.787182    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.787258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.787213    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.787783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.787762    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.789697 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.789675    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.843299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.843246    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmkh\" (UniqueName: \"kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh\") pod \"maas-api-revoke-sub-3e31a506-5lwqx\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:25:59.861951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:25:59.861929    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:26:00.235160 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:00.235119    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"]
Oct 30 21:26:00.238372 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:26:00.238341    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8d5107_3cbf_4866_9813_87e4c341dac2.slice/crio-402eb26288b94a1e00a74c73fdde7b9bff4612b3398330aef98d8d50cc31ded0 WatchSource:0}: Error finding container 402eb26288b94a1e00a74c73fdde7b9bff4612b3398330aef98d8d50cc31ded0: Status 404 returned error can't find the container with id 402eb26288b94a1e00a74c73fdde7b9bff4612b3398330aef98d8d50cc31ded0
Oct 30 21:26:00.479664 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:00.479627    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx" event={"ID":"3f8d5107-3cbf-4866-9813-87e4c341dac2","Type":"ContainerStarted","Data":"402eb26288b94a1e00a74c73fdde7b9bff4612b3398330aef98d8d50cc31ded0"}
Oct 30 21:26:01.487940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:01.487899    5967 generic.go:356] "Generic (PLEG): container finished" podID="3f8d5107-3cbf-4866-9813-87e4c341dac2" containerID="5a3ccd06436886084e322ba8e3acc1e3996a079330acaaac3509faf29b1cf16c" exitCode=0
Oct 30 21:26:01.488457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:01.487983    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx" event={"ID":"3f8d5107-3cbf-4866-9813-87e4c341dac2","Type":"ContainerDied","Data":"5a3ccd06436886084e322ba8e3acc1e3996a079330acaaac3509faf29b1cf16c"}
Oct 30 21:26:01.980952 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:26:01.980918    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:26:02.949659 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:02.949632    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:26:03.053684 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.053640    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh\" (UniqueName: \"kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh\") pod \"3f8d5107-3cbf-4866-9813-87e4c341dac2\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") "
Oct 30 21:26:03.053913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.053695    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca\") pod \"3f8d5107-3cbf-4866-9813-87e4c341dac2\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") "
Oct 30 21:26:03.053913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.053730    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca\") pod \"3f8d5107-3cbf-4866-9813-87e4c341dac2\" (UID: \"3f8d5107-3cbf-4866-9813-87e4c341dac2\") "
Oct 30 21:26:03.054061 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.054007    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca" pod "3f8d5107-3cbf-4866-9813-87e4c341dac2" (UID: "3f8d5107-3cbf-4866-9813-87e4c341dac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:03.055846 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.055822    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca" pod "3f8d5107-3cbf-4866-9813-87e4c341dac2" (UID: "3f8d5107-3cbf-4866-9813-87e4c341dac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:03.055932 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.055912    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh" pod "3f8d5107-3cbf-4866-9813-87e4c341dac2" (UID: "3f8d5107-3cbf-4866-9813-87e4c341dac2"). InnerVolumeSpecName "kube-api-access-wdmkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:03.155572 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.155472    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.155572 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.155512    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmkh\" (UniqueName: \"kubernetes.io/projected/3f8d5107-3cbf-4866-9813-87e4c341dac2-kube-api-access-wdmkh\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.155572 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.155528    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d5107-3cbf-4866-9813-87e4c341dac2-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.541080 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.541054    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"
Oct 30 21:26:03.548446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.548399    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx" event={"ID":"3f8d5107-3cbf-4866-9813-87e4c341dac2","Type":"ContainerDied","Data":"402eb26288b94a1e00a74c73fdde7b9bff4612b3398330aef98d8d50cc31ded0"}
Oct 30 21:26:03.548617 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.548454    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402eb26288b94a1e00a74c73fdde7b9bff4612b3398330aef98d8d50cc31ded0"
Oct 30 21:26:03.651361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.651324    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" podUID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" containerName="main" containerID="cri-o://b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8" gracePeriod=45
Oct 30 21:26:03.914678 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.914656    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:26:03.963818 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.963789    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724\" (UniqueName: \"kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724\") pod \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") "
Oct 30 21:26:03.964228 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.963864    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm\") pod \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") "
Oct 30 21:26:03.964228 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964001    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache\") pod \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") "
Oct 30 21:26:03.964228 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964037    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs\") pod \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") "
Oct 30 21:26:03.964228 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964065    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home\") pod \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") "
Oct 30 21:26:03.964228 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964134    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir\") pod \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\" (UID: \"9c6f22a9-c6ae-4afb-9341-5165802b8b04\") "
Oct 30 21:26:03.964666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964244    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache" pod "9c6f22a9-c6ae-4afb-9341-5165802b8b04" (UID: "9c6f22a9-c6ae-4afb-9341-5165802b8b04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.964666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964335    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home" pod "9c6f22a9-c6ae-4afb-9341-5165802b8b04" (UID: "9c6f22a9-c6ae-4afb-9341-5165802b8b04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.964666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964404    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir" pod "9c6f22a9-c6ae-4afb-9341-5165802b8b04" (UID: "9c6f22a9-c6ae-4afb-9341-5165802b8b04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.964918 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964898    5967 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-model-cache\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.964988 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964922    5967 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-home\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.964988 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.964936    5967 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tmp-dir\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.966315 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.966288    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724" pod "9c6f22a9-c6ae-4afb-9341-5165802b8b04" (UID: "9c6f22a9-c6ae-4afb-9341-5165802b8b04"). InnerVolumeSpecName "kube-api-access-2z724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:03.966417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.966316    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs" pod "9c6f22a9-c6ae-4afb-9341-5165802b8b04" (UID: "9c6f22a9-c6ae-4afb-9341-5165802b8b04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:03.966417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:03.966327    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm" pod "9c6f22a9-c6ae-4afb-9341-5165802b8b04" (UID: "9c6f22a9-c6ae-4afb-9341-5165802b8b04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:04.066579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.066543    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z724\" (UniqueName: \"kubernetes.io/projected/9c6f22a9-c6ae-4afb-9341-5165802b8b04-kube-api-access-2z724\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.066579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.066570    5967 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f22a9-c6ae-4afb-9341-5165802b8b04-dshm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.066579 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.066579    5967 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f22a9-c6ae-4afb-9341-5165802b8b04-tls-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.547241 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.547205    5967 generic.go:356] "Generic (PLEG): container finished" podID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" containerID="b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8" exitCode=0
Oct 30 21:26:04.547417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.547254    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"
Oct 30 21:26:04.547417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.547289    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" event={"ID":"9c6f22a9-c6ae-4afb-9341-5165802b8b04","Type":"ContainerDied","Data":"b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8"}
Oct 30 21:26:04.547417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.547334    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" event={"ID":"9c6f22a9-c6ae-4afb-9341-5165802b8b04","Type":"ContainerDied","Data":"f13412dac0142c1508ec9e32a853f581388eee4fb1f20f3815411ea3f6775411"}
Oct 30 21:26:04.547417 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.547360    5967 scope.go:122] "RemoveContainer" containerID="b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8"
Oct 30 21:26:04.557001 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.556983    5967 scope.go:122] "RemoveContainer" containerID="b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8"
Oct 30 21:26:04.557257 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:26:04.557239    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8\": container with ID starting with b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8 not found: ID does not exist" containerID="b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8"
Oct 30 21:26:04.557311 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.557268    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8"} err="failed to get container status \"b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8\": rpc error: code = NotFound desc = could not find container \"b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8\": container with ID starting with b17c5fabbc6a3f6b6446ef625632ed3c05916f58219cd253d81e44d365fd5ba8 not found: ID does not exist"
Oct 30 21:26:04.712279 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.712239    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"]
Oct 30 21:26:04.757783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.757752    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs"]
Oct 30 21:26:04.901346 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:04.901263    5967 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-569edc7e-kserve-86d9c797cc-n8jbs" podUID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" containerName="main" probeResult="failure" output="Get \"https://10.134.1.6:8000/ready\": context deadline exceeded"
Oct 30 21:26:05.541300 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:05.541259    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" path="/var/lib/kubelet/pods/9c6f22a9-c6ae-4afb-9341-5165802b8b04/volumes"
Oct 30 21:26:23.224591 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:23.224561    5967 scope.go:122] "RemoveContainer" containerID="081ac58bd44ad6451fd24c816de062c67df06fd09a341cc5c485e937ba958afd"
Oct 30 21:26:29.523711 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.523634    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"]
Oct 30 21:26:29.524180 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.524162    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3f8d5107-3cbf-4866-9813-87e4c341dac2" containerName="revoke-keys"
Oct 30 21:26:29.524239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.524184    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8d5107-3cbf-4866-9813-87e4c341dac2" containerName="revoke-keys"
Oct 30 21:26:29.524239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.524218    5967 cpu_manager.go:414] "containerMap: removing container" podUID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" containerName="main"
Oct 30 21:26:29.524239 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.524225    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" containerName="main"
Oct 30 21:26:29.524382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.524351    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6f22a9-c6ae-4afb-9341-5165802b8b04" containerName="main"
Oct 30 21:26:29.524382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.524372    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8d5107-3cbf-4866-9813-87e4c341dac2" containerName="revoke-keys"
Oct 30 21:26:29.527682 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.527663    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.580438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.580409    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:29.622897 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.622864    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvcm\" (UniqueName: \"kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.623074 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.623027    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.623074 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.623053    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.629674 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.629651    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:29.629832 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.629655    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:29.629832 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.629688    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:26:29.631734 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.631716    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"]
Oct 30 21:26:29.724149 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.724124    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvcm\" (UniqueName: \"kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.724299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.724211    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.724299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.724226    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.724879 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.724860    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.726531 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.726516    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.781768 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.781685    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvcm\" (UniqueName: \"kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm\") pod \"maas-api-revoke-sub-af40c647-nrttr\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:29.842547 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:29.842524    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:30.212821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:30.212798    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"]
Oct 30 21:26:30.215076 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:26:30.215044    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ac9137_f50c_4c47_a638_f3d5b170bf5f.slice/crio-3a87bd25d5e87810f0bfaebd98f42fac5738d74373d80b069992e187714afd5e WatchSource:0}: Error finding container 3a87bd25d5e87810f0bfaebd98f42fac5738d74373d80b069992e187714afd5e: Status 404 returned error can't find the container with id 3a87bd25d5e87810f0bfaebd98f42fac5738d74373d80b069992e187714afd5e
Oct 30 21:26:30.708456 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:30.708419    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr" event={"ID":"36ac9137-f50c-4c47-a638-f3d5b170bf5f","Type":"ContainerStarted","Data":"f110dd20186f20bd5eb65d211263d2f07a15acf6d11b5b755b01431a1a938511"}
Oct 30 21:26:30.708940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:30.708484    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr" event={"ID":"36ac9137-f50c-4c47-a638-f3d5b170bf5f","Type":"ContainerStarted","Data":"3a87bd25d5e87810f0bfaebd98f42fac5738d74373d80b069992e187714afd5e"}
Oct 30 21:26:30.824479 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:30.824389    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr" podStartSLOduration=1.824375141 podStartE2EDuration="1.824375141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:30.822238236 +0000 UTC m=+3971.423040199" watchObservedRunningTime="2026-10-30 21:26:30.824375141 +0000 UTC m=+3971.425177099"
Oct 30 21:26:31.715446 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:31.715413    5967 generic.go:356] "Generic (PLEG): container finished" podID="36ac9137-f50c-4c47-a638-f3d5b170bf5f" containerID="f110dd20186f20bd5eb65d211263d2f07a15acf6d11b5b755b01431a1a938511" exitCode=0
Oct 30 21:26:31.715904 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:31.715453    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr" event={"ID":"36ac9137-f50c-4c47-a638-f3d5b170bf5f","Type":"ContainerDied","Data":"f110dd20186f20bd5eb65d211263d2f07a15acf6d11b5b755b01431a1a938511"}
Oct 30 21:26:31.979549 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:26:31.979470    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:26:32.858629 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.858604    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:32.952876 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.952851    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca\") pod \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") "
Oct 30 21:26:32.952998 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.952894    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm\" (UniqueName: \"kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm\") pod \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") "
Oct 30 21:26:32.952998 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.952954    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca\") pod \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\" (UID: \"36ac9137-f50c-4c47-a638-f3d5b170bf5f\") "
Oct 30 21:26:32.953194 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.953172    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca" pod "36ac9137-f50c-4c47-a638-f3d5b170bf5f" (UID: "36ac9137-f50c-4c47-a638-f3d5b170bf5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:32.953261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.953249    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:32.955001 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.954978    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm" pod "36ac9137-f50c-4c47-a638-f3d5b170bf5f" (UID: "36ac9137-f50c-4c47-a638-f3d5b170bf5f"). InnerVolumeSpecName "kube-api-access-kmvcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:32.955149 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:32.955126    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca" pod "36ac9137-f50c-4c47-a638-f3d5b170bf5f" (UID: "36ac9137-f50c-4c47-a638-f3d5b170bf5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:33.054760 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:33.054671    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvcm\" (UniqueName: \"kubernetes.io/projected/36ac9137-f50c-4c47-a638-f3d5b170bf5f-kube-api-access-kmvcm\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:33.054760 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:33.054695    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ac9137-f50c-4c47-a638-f3d5b170bf5f-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:33.726691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:33.726662    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"
Oct 30 21:26:33.726691 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:33.726670    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr" event={"ID":"36ac9137-f50c-4c47-a638-f3d5b170bf5f","Type":"ContainerDied","Data":"3a87bd25d5e87810f0bfaebd98f42fac5738d74373d80b069992e187714afd5e"}
Oct 30 21:26:33.726909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:33.726714    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a87bd25d5e87810f0bfaebd98f42fac5738d74373d80b069992e187714afd5e"
Oct 30 21:26:37.045457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:37.045426    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"]
Oct 30 21:26:37.096402 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:37.096366    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k7gq4"]
Oct 30 21:26:37.541282 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:37.541252    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb9dfc6-c513-4b54-afa3-7f0ac40bc039" path="/var/lib/kubelet/pods/edb9dfc6-c513-4b54-afa3-7f0ac40bc039/volumes"
Oct 30 21:26:51.849184 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.849152    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"]
Oct 30 21:26:51.849652 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.849595    5967 cpu_manager.go:414] "containerMap: removing container" podUID="36ac9137-f50c-4c47-a638-f3d5b170bf5f" containerName="revoke-keys"
Oct 30 21:26:51.849652 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.849605    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ac9137-f50c-4c47-a638-f3d5b170bf5f" containerName="revoke-keys"
Oct 30 21:26:51.849730 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.849698    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ac9137-f50c-4c47-a638-f3d5b170bf5f" containerName="revoke-keys"
Oct 30 21:26:51.854091 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.854075    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.920003 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.919979    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fb4770d7-openshift-default-dockercfg-f684s\""
Oct 30 21:26:51.930395 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930369    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.930547 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930433    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.930547 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930469    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.930687 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930581    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.930811 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930787    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.930955 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930839    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9br\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.930955 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930878    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.931098 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.930973    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.931098 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.931013    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:51.958138 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:51.958108    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"]
Oct 30 21:26:52.031940 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.031894    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.031973    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032003    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9br\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032029    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032052    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032084    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032153 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032142    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032162    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032211    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032599    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.032882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.032860    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.033073 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.033045    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.033232 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.033097    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.033307 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.033276    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.034568 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.034545    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.034931 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.034915    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.091286 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.091262    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.091391 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.091290    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9br\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br\") pod \"e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") " pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.167681 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.167602    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:52.593665 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.593641    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"]
Oct 30 21:26:52.596814 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:26:52.596791    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569f2b49_cccd_409e_a132_d8c05be050ff.slice/crio-03c9d7d98b34101d2871ee18e9588da19308ea97e73749716022a7b030a6c687 WatchSource:0}: Error finding container 03c9d7d98b34101d2871ee18e9588da19308ea97e73749716022a7b030a6c687: Status 404 returned error can't find the container with id 03c9d7d98b34101d2871ee18e9588da19308ea97e73749716022a7b030a6c687
Oct 30 21:26:52.599258 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.599218    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:52.599342 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.599309    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:52.599391 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.599374    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:52.828553 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.828516    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" event={"ID":"569f2b49-cccd-409e-a132-d8c05be050ff","Type":"ContainerStarted","Data":"d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"}
Oct 30 21:26:52.828735 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.828577    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" event={"ID":"569f2b49-cccd-409e-a132-d8c05be050ff","Type":"ContainerStarted","Data":"03c9d7d98b34101d2871ee18e9588da19308ea97e73749716022a7b030a6c687"}
Oct 30 21:26:52.947255 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:52.947162    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podStartSLOduration=1.94714748 podStartE2EDuration="1.94714748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:52.944912002 +0000 UTC m=+3993.545713965" watchObservedRunningTime="2026-10-30 21:26:52.94714748 +0000 UTC m=+3993.547949437"
Oct 30 21:26:53.168241 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:53.168211    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:53.173180 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:53.173158    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:53.833304 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:53.833264    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:26:53.834351 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:26:53.834328    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:27:01.980667 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:27:01.980634    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:27:06.048466 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:06.048436    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"]
Oct 30 21:27:06.096989 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:06.096961    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qbv9l"]
Oct 30 21:27:07.541245 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:07.541214    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8d8c97-135c-49a7-b0ac-4fe281e78882" path="/var/lib/kubelet/pods/7e8d8c97-135c-49a7-b0ac-4fe281e78882/volumes"
Oct 30 21:27:16.959223 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:16.959191    5967 generic.go:356] "Generic (PLEG): container finished" podID="569f2b49-cccd-409e-a132-d8c05be050ff" containerID="d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19" exitCode=0
Oct 30 21:27:16.959638 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:16.959264    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" event={"ID":"569f2b49-cccd-409e-a132-d8c05be050ff","Type":"ContainerDied","Data":"d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"}
Oct 30 21:27:16.959638 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:16.959583    5967 scope.go:122] "RemoveContainer" containerID="d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"
Oct 30 21:27:16.960678 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:16.960653    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:16.960731 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:16.960720    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:16.960791 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:16.960781    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:17.967007 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:17.966973    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" event={"ID":"569f2b49-cccd-409e-a132-d8c05be050ff","Type":"ContainerStarted","Data":"7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867"}
Oct 30 21:27:18.168462 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:18.168429    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:27:18.183766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:18.168721    5967 patch_prober.go:29] interesting pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:27:18.183766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:18.168809    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 21:27:19.168382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:19.168349    5967 patch_prober.go:29] interesting pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:27:19.168859 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:19.168406    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 21:27:20.168592 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:20.168556    5967 patch_prober.go:29] interesting pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:27:20.169091 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:20.168608    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 21:27:21.168577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:21.168544    5967 patch_prober.go:29] interesting pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:27:21.168803 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:21.168590    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 21:27:22.168682 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:22.168646    5967 patch_prober.go:29] interesting pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:27:22.168954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:22.168703    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 21:27:23.171827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:23.171798    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:27:23.172249 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:23.172172    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:27:23.172781 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:23.172759    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:27:28.917168 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:28.917138    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"]
Oct 30 21:27:28.919874 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:28.919857    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:28.971980 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:28.971953    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:27:29.007475 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.007449    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.007599 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.007480    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm5n\" (UniqueName: \"kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.007650 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.007635    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.021379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.021347    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:27:29.021487 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.021393    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:27:29.021529 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.021499    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8nnvq\""
Oct 30 21:27:29.023538 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.023519    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"]
Oct 30 21:27:29.108613 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.108586    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.108613 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.108617    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm5n\" (UniqueName: \"kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.108885 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.108717    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.109330 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.109309    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.111060 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.111040    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.166201 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.166174    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm5n\" (UniqueName: \"kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n\") pod \"maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.231629 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.231607    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:29.598518 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:29.598497    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"]
Oct 30 21:27:29.600605 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:27:29.600575    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cc8083_8922_4696_b371_374b0310d94b.slice/crio-d05c07808e92a430e39b46ebd80af68b1cf20784cead9d250e46d367821cc1b5 WatchSource:0}: Error finding container d05c07808e92a430e39b46ebd80af68b1cf20784cead9d250e46d367821cc1b5: Status 404 returned error can't find the container with id d05c07808e92a430e39b46ebd80af68b1cf20784cead9d250e46d367821cc1b5
Oct 30 21:27:30.054051 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:30.054017    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g" event={"ID":"f5cc8083-8922-4696-b371-374b0310d94b","Type":"ContainerStarted","Data":"4eeabb940e7fa4e9a9fcc6b21710fc1bb1391c99c3d645e701368179c0ba00e2"}
Oct 30 21:27:30.054518 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:30.054079    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g" event={"ID":"f5cc8083-8922-4696-b371-374b0310d94b","Type":"ContainerStarted","Data":"d05c07808e92a430e39b46ebd80af68b1cf20784cead9d250e46d367821cc1b5"}
Oct 30 21:27:30.168504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:30.168455    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g" podStartSLOduration=2.168440635 podStartE2EDuration="2.168440635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:30.166776474 +0000 UTC m=+4030.767578437" watchObservedRunningTime="2026-10-30 21:27:30.168440635 +0000 UTC m=+4030.769242594"
Oct 30 21:27:31.060601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:31.060570    5967 generic.go:356] "Generic (PLEG): container finished" podID="f5cc8083-8922-4696-b371-374b0310d94b" containerID="4eeabb940e7fa4e9a9fcc6b21710fc1bb1391c99c3d645e701368179c0ba00e2" exitCode=0
Oct 30 21:27:31.061067 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:31.060647    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g" event={"ID":"f5cc8083-8922-4696-b371-374b0310d94b","Type":"ContainerDied","Data":"4eeabb940e7fa4e9a9fcc6b21710fc1bb1391c99c3d645e701368179c0ba00e2"}
Oct 30 21:27:31.979143 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:27:31.979094    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:27:32.198587 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.198563    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:32.342833 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.342769    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca\") pod \"f5cc8083-8922-4696-b371-374b0310d94b\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") "
Oct 30 21:27:32.342833 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.342820    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n\" (UniqueName: \"kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n\") pod \"f5cc8083-8922-4696-b371-374b0310d94b\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") "
Oct 30 21:27:32.343043 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.342876    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca\") pod \"f5cc8083-8922-4696-b371-374b0310d94b\" (UID: \"f5cc8083-8922-4696-b371-374b0310d94b\") "
Oct 30 21:27:32.343207 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.343183    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca" pod "f5cc8083-8922-4696-b371-374b0310d94b" (UID: "f5cc8083-8922-4696-b371-374b0310d94b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:32.345011 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.344987    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca" pod "f5cc8083-8922-4696-b371-374b0310d94b" (UID: "f5cc8083-8922-4696-b371-374b0310d94b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:27:32.345117 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.345060    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n" pod "f5cc8083-8922-4696-b371-374b0310d94b" (UID: "f5cc8083-8922-4696-b371-374b0310d94b"). InnerVolumeSpecName "kube-api-access-7sm5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:32.444291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.444262    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:32.444291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.444292    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sm5n\" (UniqueName: \"kubernetes.io/projected/f5cc8083-8922-4696-b371-374b0310d94b-kube-api-access-7sm5n\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:32.444451 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:32.444304    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5cc8083-8922-4696-b371-374b0310d94b-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:33.071906 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:33.071875    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"
Oct 30 21:27:33.072220 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:33.071954    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g" event={"ID":"f5cc8083-8922-4696-b371-374b0310d94b","Type":"ContainerDied","Data":"d05c07808e92a430e39b46ebd80af68b1cf20784cead9d250e46d367821cc1b5"}
Oct 30 21:27:33.072220 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:33.072003    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05c07808e92a430e39b46ebd80af68b1cf20784cead9d250e46d367821cc1b5"
Oct 30 21:27:50.462420 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:50.462384    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"]
Oct 30 21:27:50.511449 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:50.511425    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb4770d7-42j4g"]
Oct 30 21:27:51.541608 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:51.541569    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5cc8083-8922-4696-b371-374b0310d94b" path="/var/lib/kubelet/pods/f5cc8083-8922-4696-b371-374b0310d94b/volumes"
Oct 30 21:27:54.138942 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:54.138913    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"]
Oct 30 21:27:54.139406 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:54.139125    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy" containerID="cri-o://7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867" gracePeriod=30
Oct 30 21:27:56.018769 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.018711    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"]
Oct 30 21:27:56.019205 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.019191    5967 cpu_manager.go:414] "containerMap: removing container" podUID="f5cc8083-8922-4696-b371-374b0310d94b" containerName="revoke-keys"
Oct 30 21:27:56.019261 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.019206    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5cc8083-8922-4696-b371-374b0310d94b" containerName="revoke-keys"
Oct 30 21:27:56.019304 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.019275    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5cc8083-8922-4696-b371-374b0310d94b" containerName="revoke-keys"
Oct 30 21:27:56.022455 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.022436    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065231 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065195    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065413 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065252    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065413 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065329    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065413 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065385    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065462    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065491    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065547    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065601 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065581    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcmz\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.065860 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.065617    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.109722 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.109697    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-0a3a62-openshift-default-dockercfg-6qjqf\""
Oct 30 21:27:56.126254 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.126229    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"]
Oct 30 21:27:56.166530 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166493    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166548    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166602    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166618    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166667 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166657    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166921 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166688    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcmz\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166921 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166718    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166921 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166785    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.166921 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.166813    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.167221 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.167182    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.167512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.167234    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.167512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.167252    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.167512 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.167396    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.167719 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.167577    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.169098 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.169081    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.169204 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.169124    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.223702 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.223666    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.223909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.223771    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcmz\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz\") pod \"e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") " pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.336165 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.336090    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:56.763533 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.763505    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"]
Oct 30 21:27:56.765398 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:27:56.765376    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c76bb41_6580_4521_99f4_d6fc1bff12b6.slice/crio-53ca934958f95dd9fa2c523cd93a8c31537600f9356296d6fb81eb0c3cc6298f WatchSource:0}: Error finding container 53ca934958f95dd9fa2c523cd93a8c31537600f9356296d6fb81eb0c3cc6298f: Status 404 returned error can't find the container with id 53ca934958f95dd9fa2c523cd93a8c31537600f9356296d6fb81eb0c3cc6298f
Oct 30 21:27:56.767535 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.767499    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:56.767636 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.767576    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:56.767636 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:56.767619    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:57.196806 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:57.196693    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" event={"ID":"3c76bb41-6580-4521-99f4-d6fc1bff12b6","Type":"ContainerStarted","Data":"8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"}
Oct 30 21:27:57.196806 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:57.196777    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" event={"ID":"3c76bb41-6580-4521-99f4-d6fc1bff12b6","Type":"ContainerStarted","Data":"53ca934958f95dd9fa2c523cd93a8c31537600f9356296d6fb81eb0c3cc6298f"}
Oct 30 21:27:57.336360 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:57.336322    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:57.341289 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:57.341265    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:57.459049 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:57.458949    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" podStartSLOduration=2.458935856 podStartE2EDuration="2.458935856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:57.318692754 +0000 UTC m=+4057.919494713" watchObservedRunningTime="2026-10-30 21:27:57.458935856 +0000 UTC m=+4058.059737878"
Oct 30 21:27:58.202510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:58.202483    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:58.203369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:58.203349    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:27:59.397796 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.397766    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:27:59.500625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500596    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500633    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500663    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500801 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500764    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500799    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500850    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500882    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.500979 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500925    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.501197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.500962    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:59.501197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501005    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token\") pod \"569f2b49-cccd-409e-a132-d8c05be050ff\" (UID: \"569f2b49-cccd-409e-a132-d8c05be050ff\") "
Oct 30 21:27:59.501197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501044    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:59.501197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501055    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:59.501197 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501091    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:59.501418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501245    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:59.501418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501359    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f2b49-cccd-409e-a132-d8c05be050ff-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.501418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501372    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.501418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501381    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.501418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501389    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.501418 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.501398    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.503173 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.503147    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "kube-api-access-tt9br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:59.503173 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.503159    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:59.503320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.503185    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:59.503320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.503240    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo" pod "569f2b49-cccd-409e-a132-d8c05be050ff" (UID: "569f2b49-cccd-409e-a132-d8c05be050ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:27:59.602382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.602354    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f2b49-cccd-409e-a132-d8c05be050ff-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.602382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.602383    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt9br\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-kube-api-access-tt9br\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.602623 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.602397    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f2b49-cccd-409e-a132-d8c05be050ff-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:59.602623 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:27:59.602410    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f2b49-cccd-409e-a132-d8c05be050ff-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:28:00.214570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.214536    5967 generic.go:356] "Generic (PLEG): container finished" podID="569f2b49-cccd-409e-a132-d8c05be050ff" containerID="7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867" exitCode=0
Oct 30 21:28:00.214860 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.214586    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" event={"ID":"569f2b49-cccd-409e-a132-d8c05be050ff","Type":"ContainerDied","Data":"7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867"}
Oct 30 21:28:00.214860 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.214645    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2" event={"ID":"569f2b49-cccd-409e-a132-d8c05be050ff","Type":"ContainerDied","Data":"03c9d7d98b34101d2871ee18e9588da19308ea97e73749716022a7b030a6c687"}
Oct 30 21:28:00.214860 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.214596    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"
Oct 30 21:28:00.214860 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.214669    5967 scope.go:122] "RemoveContainer" containerID="7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867"
Oct 30 21:28:00.225403 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.225384    5967 scope.go:122] "RemoveContainer" containerID="d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"
Oct 30 21:28:00.235024 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.235006    5967 scope.go:122] "RemoveContainer" containerID="7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867"
Oct 30 21:28:00.235314 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:28:00.235291    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867\": container with ID starting with 7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867 not found: ID does not exist" containerID="7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867"
Oct 30 21:28:00.235367 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.235326    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867"} err="failed to get container status \"7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867\": rpc error: code = NotFound desc = could not find container \"7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867\": container with ID starting with 7b27a688dccb29a86dc964f9fc5ecb1a305a85b7c9e00f7ee243a3022489b867 not found: ID does not exist"
Oct 30 21:28:00.235367 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.235349    5967 scope.go:122] "RemoveContainer" containerID="d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"
Oct 30 21:28:00.235640 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:28:00.235622    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19\": container with ID starting with d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19 not found: ID does not exist" containerID="d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"
Oct 30 21:28:00.235710 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.235647    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19"} err="failed to get container status \"d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19\": rpc error: code = NotFound desc = could not find container \"d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19\": container with ID starting with d05398327c6d79ec4d7d160aa694baebb5de8fecd1fe8a4cd0570804579b3e19 not found: ID does not exist"
Oct 30 21:28:00.430532 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.430491    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"]
Oct 30 21:28:00.479523 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:00.479453    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2"]
Oct 30 21:28:01.541580 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:01.541546    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" path="/var/lib/kubelet/pods/569f2b49-cccd-409e-a132-d8c05be050ff/volumes"
Oct 30 21:28:01.980385 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:28:01.980353    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:28:22.353088 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:22.353052    5967 generic.go:356] "Generic (PLEG): container finished" podID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerID="8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540" exitCode=0
Oct 30 21:28:22.353526 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:22.353111    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" event={"ID":"3c76bb41-6580-4521-99f4-d6fc1bff12b6","Type":"ContainerDied","Data":"8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"}
Oct 30 21:28:22.353526 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:22.353522    5967 scope.go:122] "RemoveContainer" containerID="8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"
Oct 30 21:28:22.354721 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:22.354693    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:22.354843 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:22.354776    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:22.354843 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:22.354826    5967 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:23.258685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:23.258651    5967 scope.go:122] "RemoveContainer" containerID="a3a12fdc266c994dbe5a10be750b461f9d934d702744ccbba33347c0ca61d0bd"
Oct 30 21:28:23.267546 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:23.267526    5967 scope.go:122] "RemoveContainer" containerID="dedc25cbaf29a4839a29e68e37f967b2343531cc8ea4f1662c8f529fdf155470"
Oct 30 21:28:23.359762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:23.359710    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" event={"ID":"3c76bb41-6580-4521-99f4-d6fc1bff12b6","Type":"ContainerStarted","Data":"65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948"}
Oct 30 21:28:24.337223 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:24.337192    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:28:24.338209 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:24.338178    5967 patch_prober.go:29] interesting pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 21:28:24.338355 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:24.338236    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 21:28:25.336617 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:25.336580    5967 patch_prober.go:29] interesting pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 21:28:25.337034 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:25.336647    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 21:28:26.337535 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:26.337500    5967 patch_prober.go:29] interesting pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 21:28:26.337998 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:26.337550    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 21:28:27.337380 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:27.337348    5967 patch_prober.go:29] interesting pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 21:28:27.337648 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:27.337403    5967 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 21:28:28.203382 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:28.203343    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:28:28.340932 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:28.340906    5967 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:28:28.404749 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:28.404712    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:28:31.983048 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:28:31.983014    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:28:38.052508 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:38.052477    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"]
Oct 30 21:28:38.101756 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:38.101705    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac991e0-tm9tt"]
Oct 30 21:28:39.541656 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:28:39.541626    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024a8d09-1293-44ca-839d-3ccd0573cada" path="/var/lib/kubelet/pods/024a8d09-1293-44ca-839d-3ccd0573cada/volumes"
Oct 30 21:29:01.979364 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:29:01.979332    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:29:31.980278 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:29:31.980181    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:29:45.609325 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.609292    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"]
Oct 30 21:29:45.609819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.609722    5967 cpu_manager.go:414] "containerMap: removing container" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy"
Oct 30 21:29:45.609819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.609732    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy"
Oct 30 21:29:45.609819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.609797    5967 cpu_manager.go:414] "containerMap: removing container" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy"
Oct 30 21:29:45.609819 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.609802    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy"
Oct 30 21:29:45.609965 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.609872    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy"
Oct 30 21:29:45.613095 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.613077    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.669932 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.669904    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"]
Oct 30 21:29:45.716553 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.716522    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:45.716553 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.716536    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:45.716553 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.716522    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:45.716808 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.716589    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8nnvq\""
Oct 30 21:29:45.750958 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.750925    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.751120 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.750962    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4pg\" (UniqueName: \"kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.751120 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.751084    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.852312 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.852286    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.852505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.852316    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4pg\" (UniqueName: \"kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.852505 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.852365    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.852936 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.852916    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.854826 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.854802    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.908223 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.908174    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4pg\" (UniqueName: \"kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:45.926052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:45.926034    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:46.298330 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:46.298305    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"]
Oct 30 21:29:46.300684 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:29:46.300658    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51279b0b_e730_4046_abea_e13c0468dfeb.slice/crio-d010a8390b59fe47541d53fae77fed4ef6e0b6d5718cb00de0a5dd6cf20906cb WatchSource:0}: Error finding container d010a8390b59fe47541d53fae77fed4ef6e0b6d5718cb00de0a5dd6cf20906cb: Status 404 returned error can't find the container with id d010a8390b59fe47541d53fae77fed4ef6e0b6d5718cb00de0a5dd6cf20906cb
Oct 30 21:29:46.807534 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:46.807500    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw" event={"ID":"51279b0b-e730-4046-abea-e13c0468dfeb","Type":"ContainerStarted","Data":"4ba3e1076ecde943f15e5c1f37e0a867b6a8ee9d6aec21e1a00d3c894efae049"}
Oct 30 21:29:46.807928 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:46.807557    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw" event={"ID":"51279b0b-e730-4046-abea-e13c0468dfeb","Type":"ContainerStarted","Data":"d010a8390b59fe47541d53fae77fed4ef6e0b6d5718cb00de0a5dd6cf20906cb"}
Oct 30 21:29:46.921069 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:46.920964    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw" podStartSLOduration=1.920924849 podStartE2EDuration="1.920924849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:46.919938067 +0000 UTC m=+4167.520740024" watchObservedRunningTime="2026-10-30 21:29:46.920924849 +0000 UTC m=+4167.521726809"
Oct 30 21:29:47.814549 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:47.814508    5967 generic.go:356] "Generic (PLEG): container finished" podID="51279b0b-e730-4046-abea-e13c0468dfeb" containerID="4ba3e1076ecde943f15e5c1f37e0a867b6a8ee9d6aec21e1a00d3c894efae049" exitCode=0
Oct 30 21:29:47.814971 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:47.814554    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw" event={"ID":"51279b0b-e730-4046-abea-e13c0468dfeb","Type":"ContainerDied","Data":"4ba3e1076ecde943f15e5c1f37e0a867b6a8ee9d6aec21e1a00d3c894efae049"}
Oct 30 21:29:48.964837 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:48.964815    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:49.086341 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.086271    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca\") pod \"51279b0b-e730-4046-abea-e13c0468dfeb\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") "
Oct 30 21:29:49.086504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.086358    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca\") pod \"51279b0b-e730-4046-abea-e13c0468dfeb\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") "
Oct 30 21:29:49.086504 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.086479    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg\" (UniqueName: \"kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg\") pod \"51279b0b-e730-4046-abea-e13c0468dfeb\" (UID: \"51279b0b-e730-4046-abea-e13c0468dfeb\") "
Oct 30 21:29:49.086625 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.086601    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca" pod "51279b0b-e730-4046-abea-e13c0468dfeb" (UID: "51279b0b-e730-4046-abea-e13c0468dfeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:49.086855 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.086836    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:49.088560 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.088534    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca" pod "51279b0b-e730-4046-abea-e13c0468dfeb" (UID: "51279b0b-e730-4046-abea-e13c0468dfeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:49.088911 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.088892    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg" pod "51279b0b-e730-4046-abea-e13c0468dfeb" (UID: "51279b0b-e730-4046-abea-e13c0468dfeb"). InnerVolumeSpecName "kube-api-access-zk4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:49.187730 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.187699    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4pg\" (UniqueName: \"kubernetes.io/projected/51279b0b-e730-4046-abea-e13c0468dfeb-kube-api-access-zk4pg\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:49.187730 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.187725    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51279b0b-e730-4046-abea-e13c0468dfeb-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:49.826590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.826524    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"
Oct 30 21:29:49.826751 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.826528    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw" event={"ID":"51279b0b-e730-4046-abea-e13c0468dfeb","Type":"ContainerDied","Data":"d010a8390b59fe47541d53fae77fed4ef6e0b6d5718cb00de0a5dd6cf20906cb"}
Oct 30 21:29:49.826751 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:29:49.826656    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d010a8390b59fe47541d53fae77fed4ef6e0b6d5718cb00de0a5dd6cf20906cb"
Oct 30 21:30:00.174937 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.174896    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"]
Oct 30 21:30:00.175421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.175398    5967 cpu_manager.go:414] "containerMap: removing container" podUID="51279b0b-e730-4046-abea-e13c0468dfeb" containerName="revoke-keys"
Oct 30 21:30:00.175421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.175411    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51279b0b-e730-4046-abea-e13c0468dfeb" containerName="revoke-keys"
Oct 30 21:30:00.175540 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.175505    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="51279b0b-e730-4046-abea-e13c0468dfeb" containerName="revoke-keys"
Oct 30 21:30:00.175540 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.175516    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="569f2b49-cccd-409e-a132-d8c05be050ff" containerName="istio-proxy"
Oct 30 21:30:00.178560 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.178545    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"]
Oct 30 21:30:00.178672 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.178655    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.181974 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.181946    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.232493 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.232258    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8nnvq\""
Oct 30 21:30:00.235163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.235139    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"]
Oct 30 21:30:00.268256 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.268234    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:00.268256 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.268253    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:00.284677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.284645    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"]
Oct 30 21:30:00.299276 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.299245    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.299488 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.299468    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22qv\" (UniqueName: \"kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv\") pod \"maas-api-key-cleanup-29889930-2svhd\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.299636 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.299613    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-2svhd\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.299796 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.299656    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznw6\" (UniqueName: \"kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.400332 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.400291    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.400537 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.400341    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22qv\" (UniqueName: \"kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv\") pod \"maas-api-key-cleanup-29889930-2svhd\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.400537 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.400375    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-2svhd\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.400537 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.400403    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznw6\" (UniqueName: \"kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.401058 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.401032    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.401186 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.401162    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-2svhd\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.456413 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.456352    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22qv\" (UniqueName: \"kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv\") pod \"maas-api-key-cleanup-29889930-2svhd\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.456669 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.456651    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznw6\" (UniqueName: \"kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.492233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.492207    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:00.497794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.497776    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:00.970180 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.970155    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"]
Oct 30 21:30:00.972174 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:30:00.972148    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186934a5_b0ef_49a7_8582_139757240d7e.slice/crio-0f6549e9b0a708b53ead37e1d170f07a4a1ecf2e0161c7ff48076326326cca25 WatchSource:0}: Error finding container 0f6549e9b0a708b53ead37e1d170f07a4a1ecf2e0161c7ff48076326326cca25: Status 404 returned error can't find the container with id 0f6549e9b0a708b53ead37e1d170f07a4a1ecf2e0161c7ff48076326326cca25
Oct 30 21:30:00.979519 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:00.979498    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"]
Oct 30 21:30:00.982048 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:30:00.982027    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b6a63d_8e66_44a4_946e_034a00fd54bc.slice/crio-1b594ac074a03c382ff23579b9662f9f5cdeb48d104a04c76c538907e8d3a678 WatchSource:0}: Error finding container 1b594ac074a03c382ff23579b9662f9f5cdeb48d104a04c76c538907e8d3a678: Status 404 returned error can't find the container with id 1b594ac074a03c382ff23579b9662f9f5cdeb48d104a04c76c538907e8d3a678
Oct 30 21:30:01.890380 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.890348    5967 generic.go:356] "Generic (PLEG): container finished" podID="186934a5-b0ef-49a7-8582-139757240d7e" containerID="0b14d3500f2e668dc1167986e93d04d95a915ed2aed1572d81e8321a1f68b6aa" exitCode=60
Oct 30 21:30:01.890757 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.890426    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" event={"ID":"186934a5-b0ef-49a7-8582-139757240d7e","Type":"ContainerDied","Data":"0b14d3500f2e668dc1167986e93d04d95a915ed2aed1572d81e8321a1f68b6aa"}
Oct 30 21:30:01.890757 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.890480    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" event={"ID":"186934a5-b0ef-49a7-8582-139757240d7e","Type":"ContainerStarted","Data":"0f6549e9b0a708b53ead37e1d170f07a4a1ecf2e0161c7ff48076326326cca25"}
Oct 30 21:30:01.890757 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.890685    5967 scope.go:122] "RemoveContainer" containerID="0b14d3500f2e668dc1167986e93d04d95a915ed2aed1572d81e8321a1f68b6aa"
Oct 30 21:30:01.892037 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.892020    5967 generic.go:356] "Generic (PLEG): container finished" podID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerID="240d2de885c9b055fb80897d874eb174228b5dbf0cc536f32a42bbb3b20c5d9e" exitCode=60
Oct 30 21:30:01.892113 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.892093    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w" event={"ID":"47b6a63d-8e66-44a4-946e-034a00fd54bc","Type":"ContainerDied","Data":"240d2de885c9b055fb80897d874eb174228b5dbf0cc536f32a42bbb3b20c5d9e"}
Oct 30 21:30:01.892150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.892134    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w" event={"ID":"47b6a63d-8e66-44a4-946e-034a00fd54bc","Type":"ContainerStarted","Data":"1b594ac074a03c382ff23579b9662f9f5cdeb48d104a04c76c538907e8d3a678"}
Oct 30 21:30:01.892254 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:01.892240    5967 scope.go:122] "RemoveContainer" containerID="240d2de885c9b055fb80897d874eb174228b5dbf0cc536f32a42bbb3b20c5d9e"
Oct 30 21:30:01.984361 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:01.984337    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:30:02.899367 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.899274    5967 generic.go:356] "Generic (PLEG): container finished" podID="186934a5-b0ef-49a7-8582-139757240d7e" containerID="52b298ee45bd8fe39b85a2a4dcda7dbffb126a4db676a131389bf23d75b56bca" exitCode=60
Oct 30 21:30:02.899822 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.899361    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" event={"ID":"186934a5-b0ef-49a7-8582-139757240d7e","Type":"ContainerDied","Data":"52b298ee45bd8fe39b85a2a4dcda7dbffb126a4db676a131389bf23d75b56bca"}
Oct 30 21:30:02.899822 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.899422    5967 scope.go:122] "RemoveContainer" containerID="0b14d3500f2e668dc1167986e93d04d95a915ed2aed1572d81e8321a1f68b6aa"
Oct 30 21:30:02.899822 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.899684    5967 scope.go:122] "RemoveContainer" containerID="52b298ee45bd8fe39b85a2a4dcda7dbffb126a4db676a131389bf23d75b56bca"
Oct 30 21:30:02.899991 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:02.899970    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-2svhd_odh-ai-gateway-infra(186934a5-b0ef-49a7-8582-139757240d7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" podUID="186934a5-b0ef-49a7-8582-139757240d7e"
Oct 30 21:30:02.901390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.901371    5967 generic.go:356] "Generic (PLEG): container finished" podID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerID="55c97beb16f1fcda5985899c17e7ed1119bae61c0e3fe6d521a5faedbf35cd32" exitCode=60
Oct 30 21:30:02.901497 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.901416    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w" event={"ID":"47b6a63d-8e66-44a4-946e-034a00fd54bc","Type":"ContainerDied","Data":"55c97beb16f1fcda5985899c17e7ed1119bae61c0e3fe6d521a5faedbf35cd32"}
Oct 30 21:30:02.901661 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.901646    5967 scope.go:122] "RemoveContainer" containerID="55c97beb16f1fcda5985899c17e7ed1119bae61c0e3fe6d521a5faedbf35cd32"
Oct 30 21:30:02.901896 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:02.901877    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w_odh-ai-gateway-infra(47b6a63d-8e66-44a4-946e-034a00fd54bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc"
Oct 30 21:30:02.911481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:02.911459    5967 scope.go:122] "RemoveContainer" containerID="240d2de885c9b055fb80897d874eb174228b5dbf0cc536f32a42bbb3b20c5d9e"
Oct 30 21:30:03.908648 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:03.908624    5967 scope.go:122] "RemoveContainer" containerID="52b298ee45bd8fe39b85a2a4dcda7dbffb126a4db676a131389bf23d75b56bca"
Oct 30 21:30:03.909123 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:03.908907    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-2svhd_odh-ai-gateway-infra(186934a5-b0ef-49a7-8582-139757240d7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" podUID="186934a5-b0ef-49a7-8582-139757240d7e"
Oct 30 21:30:03.910237 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:03.910217    5967 scope.go:122] "RemoveContainer" containerID="55c97beb16f1fcda5985899c17e7ed1119bae61c0e3fe6d521a5faedbf35cd32"
Oct 30 21:30:03.910445 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:03.910424    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w_odh-ai-gateway-infra(47b6a63d-8e66-44a4-946e-034a00fd54bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc"
Oct 30 21:30:07.882820 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:07.882783    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"]
Oct 30 21:30:08.025485 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.025463    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:08.070648 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.069811    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca\") pod \"47b6a63d-8e66-44a4-946e-034a00fd54bc\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") "
Oct 30 21:30:08.070648 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.069878    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6\" (UniqueName: \"kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6\") pod \"47b6a63d-8e66-44a4-946e-034a00fd54bc\" (UID: \"47b6a63d-8e66-44a4-946e-034a00fd54bc\") "
Oct 30 21:30:08.071084 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.071055    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca" pod "47b6a63d-8e66-44a4-946e-034a00fd54bc" (UID: "47b6a63d-8e66-44a4-946e-034a00fd54bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:08.074580 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.074551    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6" pod "47b6a63d-8e66-44a4-946e-034a00fd54bc" (UID: "47b6a63d-8e66-44a4-946e-034a00fd54bc"). InnerVolumeSpecName "kube-api-access-vznw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:08.170877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.170816    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b6a63d-8e66-44a4-946e-034a00fd54bc-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:08.170877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.170840    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznw6\" (UniqueName: \"kubernetes.io/projected/47b6a63d-8e66-44a4-946e-034a00fd54bc-kube-api-access-vznw6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:08.936236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.936207    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"
Oct 30 21:30:08.936684 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.936236    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w" event={"ID":"47b6a63d-8e66-44a4-946e-034a00fd54bc","Type":"ContainerDied","Data":"1b594ac074a03c382ff23579b9662f9f5cdeb48d104a04c76c538907e8d3a678"}
Oct 30 21:30:08.936684 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:08.936295    5967 scope.go:122] "RemoveContainer" containerID="55c97beb16f1fcda5985899c17e7ed1119bae61c0e3fe6d521a5faedbf35cd32"
Oct 30 21:30:09.097951 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:09.097919    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"]
Oct 30 21:30:09.147065 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:09.147019    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0a3a62-29889930-mjw2w"]
Oct 30 21:30:09.541390 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:09.541359    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" path="/var/lib/kubelet/pods/47b6a63d-8e66-44a4-946e-034a00fd54bc/volumes"
Oct 30 21:30:09.934602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:09.934526    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"]
Oct 30 21:30:09.984454 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:09.984422    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0a3a62-86gfw"]
Oct 30 21:30:11.542400 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:11.542370    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51279b0b-e730-4046-abea-e13c0468dfeb" path="/var/lib/kubelet/pods/51279b0b-e730-4046-abea-e13c0468dfeb/volumes"
Oct 30 21:30:14.535444 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:14.535415    5967 scope.go:122] "RemoveContainer" containerID="52b298ee45bd8fe39b85a2a4dcda7dbffb126a4db676a131389bf23d75b56bca"
Oct 30 21:30:14.970674 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:14.970648    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" event={"ID":"186934a5-b0ef-49a7-8582-139757240d7e","Type":"ContainerStarted","Data":"0f6ee5041661e3c533f77ccaa70f35c9d7774e343579f338a7a753fcf0cc8cb5"}
Oct 30 21:30:15.086666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:15.086589    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" podStartSLOduration=15.086575979 podStartE2EDuration="15.086575979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:15.08508521 +0000 UTC m=+4195.685887168" watchObservedRunningTime="2026-10-30 21:30:15.086575979 +0000 UTC m=+4195.687377939"
Oct 30 21:30:15.977707 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:15.977678    5967 generic.go:356] "Generic (PLEG): container finished" podID="186934a5-b0ef-49a7-8582-139757240d7e" containerID="0f6ee5041661e3c533f77ccaa70f35c9d7774e343579f338a7a753fcf0cc8cb5" exitCode=60
Oct 30 21:30:15.978267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:15.977786    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" event={"ID":"186934a5-b0ef-49a7-8582-139757240d7e","Type":"ContainerDied","Data":"0f6ee5041661e3c533f77ccaa70f35c9d7774e343579f338a7a753fcf0cc8cb5"}
Oct 30 21:30:15.978267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:15.977942    5967 scope.go:122] "RemoveContainer" containerID="52b298ee45bd8fe39b85a2a4dcda7dbffb126a4db676a131389bf23d75b56bca"
Oct 30 21:30:15.978267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:15.978201    5967 scope.go:122] "RemoveContainer" containerID="0f6ee5041661e3c533f77ccaa70f35c9d7774e343579f338a7a753fcf0cc8cb5"
Oct 30 21:30:15.978479 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:15.978456    5967 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-2svhd_odh-ai-gateway-infra(186934a5-b0ef-49a7-8582-139757240d7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" podUID="186934a5-b0ef-49a7-8582-139757240d7e"
Oct 30 21:30:16.101550 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:16.101521    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"]
Oct 30 21:30:16.631458 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:16.631427    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"]
Oct 30 21:30:16.631637 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:16.631624    5967 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy" containerID="cri-o://65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948" gracePeriod=30
Oct 30 21:30:17.120577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.120550    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:17.265255 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.265173    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca\") pod \"186934a5-b0ef-49a7-8582-139757240d7e\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") "
Oct 30 21:30:17.265255 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.265231    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv\" (UniqueName: \"kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv\") pod \"186934a5-b0ef-49a7-8582-139757240d7e\" (UID: \"186934a5-b0ef-49a7-8582-139757240d7e\") "
Oct 30 21:30:17.265619 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.265592    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca" pod "186934a5-b0ef-49a7-8582-139757240d7e" (UID: "186934a5-b0ef-49a7-8582-139757240d7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:17.267590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.267557    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv" pod "186934a5-b0ef-49a7-8582-139757240d7e" (UID: "186934a5-b0ef-49a7-8582-139757240d7e"). InnerVolumeSpecName "kube-api-access-m22qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:17.367121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.367091    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/186934a5-b0ef-49a7-8582-139757240d7e-maas-api-service-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:17.367121 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.367119    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m22qv\" (UniqueName: \"kubernetes.io/projected/186934a5-b0ef-49a7-8582-139757240d7e-kube-api-access-m22qv\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:17.990920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.990880    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd" event={"ID":"186934a5-b0ef-49a7-8582-139757240d7e","Type":"ContainerDied","Data":"0f6549e9b0a708b53ead37e1d170f07a4a1ecf2e0161c7ff48076326326cca25"}
Oct 30 21:30:17.990920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.990907    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"
Oct 30 21:30:17.991148 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:17.990948    5967 scope.go:122] "RemoveContainer" containerID="0f6ee5041661e3c533f77ccaa70f35c9d7774e343579f338a7a753fcf0cc8cb5"
Oct 30 21:30:18.153294 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:18.153261    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"]
Oct 30 21:30:18.202306 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:18.202281    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-2svhd"]
Oct 30 21:30:19.543877 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:19.543847    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186934a5-b0ef-49a7-8582-139757240d7e" path="/var/lib/kubelet/pods/186934a5-b0ef-49a7-8582-139757240d7e/volumes"
Oct 30 21:30:21.876095 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:21.876070    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:30:22.011823 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.011788    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.011835    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.011863    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.011922    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.011948    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012026 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.011972    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012030    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012092    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012171    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:22.012320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012182    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token\") pod \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\" (UID: \"3c76bb41-6580-4521-99f4-d6fc1bff12b6\") "
Oct 30 21:30:22.012320 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012296    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:22.012603 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012325    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:22.012603 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012357    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.012713 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012618    5967 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istiod-ca-cert\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.012713 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012637    5967 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-certs\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.012713 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012651    5967 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-workload-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.012918 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.012901    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:22.014756 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.014712    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "kube-api-access-zlcmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.015105 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.015071    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.015206 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.015101    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:22.015282 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.015263    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo" pod "3c76bb41-6580-4521-99f4-d6fc1bff12b6" (UID: "3c76bb41-6580-4521-99f4-d6fc1bff12b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:22.017611 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.017588    5967 generic.go:356] "Generic (PLEG): container finished" podID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerID="65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948" exitCode=0
Oct 30 21:30:22.017679 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.017636    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"
Oct 30 21:30:22.017679 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.017667    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" event={"ID":"3c76bb41-6580-4521-99f4-d6fc1bff12b6","Type":"ContainerDied","Data":"65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948"}
Oct 30 21:30:22.017799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.017717    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn" event={"ID":"3c76bb41-6580-4521-99f4-d6fc1bff12b6","Type":"ContainerDied","Data":"53ca934958f95dd9fa2c523cd93a8c31537600f9356296d6fb81eb0c3cc6298f"}
Oct 30 21:30:22.017799 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.017761    5967 scope.go:122] "RemoveContainer" containerID="65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948"
Oct 30 21:30:22.037067 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.037052    5967 scope.go:122] "RemoveContainer" containerID="8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"
Oct 30 21:30:22.045266 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.045250    5967 scope.go:122] "RemoveContainer" containerID="65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948"
Oct 30 21:30:22.045791 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:22.045765    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948\": container with ID starting with 65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948 not found: ID does not exist" containerID="65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948"
Oct 30 21:30:22.045882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.045795    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948"} err="failed to get container status \"65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948\": rpc error: code = NotFound desc = could not find container \"65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948\": container with ID starting with 65bdbcc1cfa1e343fbe1ccc4985ed3a1089b741a39cb12f8b6e1188cda218948 not found: ID does not exist"
Oct 30 21:30:22.045882 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.045818    5967 scope.go:122] "RemoveContainer" containerID="8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"
Oct 30 21:30:22.046199 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:22.046181    5967 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540\": container with ID starting with 8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540 not found: ID does not exist" containerID="8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"
Oct 30 21:30:22.046285 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.046205    5967 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540"} err="failed to get container status \"8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540\": rpc error: code = NotFound desc = could not find container \"8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540\": container with ID starting with 8cdda2b1029e5da551a325cb2b9286af532e095e279eb9b72707c9c4ade8e540 not found: ID does not exist"
Oct 30 21:30:22.113883 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.113855    5967 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-token\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.113883 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.113878    5967 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-credential-socket\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.113883 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.113887    5967 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-podinfo\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.114079 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.113896    5967 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-data\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.114079 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.113905    5967 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c76bb41-6580-4521-99f4-d6fc1bff12b6-istio-envoy\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.114079 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.113914    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlcmz\" (UniqueName: \"kubernetes.io/projected/3c76bb41-6580-4521-99f4-d6fc1bff12b6-kube-api-access-zlcmz\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.156050 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.156022    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"]
Oct 30 21:30:22.159199 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:22.159177    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn"]
Oct 30 21:30:23.331897 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:23.331868    5967 scope.go:122] "RemoveContainer" containerID="c107ae9013b69ee68b861c3159510109060fae7eed3d8a37b5b4364e177e73b2"
Oct 30 21:30:23.343081 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:23.343056    5967 scope.go:122] "RemoveContainer" containerID="319e6cc615ce497c47a428377a9ec9e881205a0e695ae6c23f6f17134190a49c"
Oct 30 21:30:23.543438 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:23.543400    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" path="/var/lib/kubelet/pods/3c76bb41-6580-4521-99f4-d6fc1bff12b6/volumes"
Oct 30 21:30:31.978496 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:30:31.978464    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:30:40.046120 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.046082    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"]
Oct 30 21:30:40.095100 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.095071    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wltdm"]
Oct 30 21:30:40.722265 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722233    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"]
Oct 30 21:30:40.722765 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722720    5967 cpu_manager.go:414] "containerMap: removing container" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:30:40.722765 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722757    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722771    5967 cpu_manager.go:414] "containerMap: removing container" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722778    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722829    5967 cpu_manager.go:414] "containerMap: removing container" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerName="cleanup"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722836    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerName="cleanup"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722857    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722863    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722877    5967 cpu_manager.go:414] "containerMap: removing container" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722884    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722897    5967 cpu_manager.go:414] "containerMap: removing container" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerName="cleanup"
Oct 30 21:30:40.722946 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.722903    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerName="cleanup"
Oct 30 21:30:40.723447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.723017    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:30:40.723447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.723030    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerName="cleanup"
Oct 30 21:30:40.723447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.723042    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy"
Oct 30 21:30:40.723447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.723051    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c76bb41-6580-4521-99f4-d6fc1bff12b6" containerName="istio-proxy"
Oct 30 21:30:40.723447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.723063    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b6a63d-8e66-44a4-946e-034a00fd54bc" containerName="cleanup"
Oct 30 21:30:40.723447 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.723074    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:30:40.726078 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.726058    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.777803 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.777781    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:40.777906 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.777847    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:40.795424 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.795395    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.795562 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.795450    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.795642 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.795570    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8c9\" (UniqueName: \"kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.827358 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.827335    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8nnvq\""
Oct 30 21:30:40.827484 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.827371    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:40.829048 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.829030    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"]
Oct 30 21:30:40.896652 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.896622    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8c9\" (UniqueName: \"kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.896787 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.896708    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.896845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.896819    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.897487 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.897471    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.899264 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.899239    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:40.953681 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:40.953658    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8c9\" (UniqueName: \"kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9\") pod \"maas-api-revoke-sub-f03629db-79mxj\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:41.039154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:41.039098    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:41.416980 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:41.416900    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"]
Oct 30 21:30:41.419390 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:30:41.419352    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f456893_663b_4a81_af12_6cca81861cc8.slice/crio-8a7ab2efcf977cd5c0a29a4ec4c2b6dc6c460718cdb83ec7384c415d1ee67546 WatchSource:0}: Error finding container 8a7ab2efcf977cd5c0a29a4ec4c2b6dc6c460718cdb83ec7384c415d1ee67546: Status 404 returned error can't find the container with id 8a7ab2efcf977cd5c0a29a4ec4c2b6dc6c460718cdb83ec7384c415d1ee67546
Oct 30 21:30:41.541806 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:41.541780    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04172ef-0ae4-4fe1-9734-500abe12c0e8" path="/var/lib/kubelet/pods/f04172ef-0ae4-4fe1-9734-500abe12c0e8/volumes"
Oct 30 21:30:42.162939 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:42.162905    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj" event={"ID":"2f456893-663b-4a81-af12-6cca81861cc8","Type":"ContainerStarted","Data":"dbccb868dbc82ea425eb09e8fe3b6067855ec51ed2f033d3d9b92bf2c85a53fd"}
Oct 30 21:30:42.163106 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:42.162965    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj" event={"ID":"2f456893-663b-4a81-af12-6cca81861cc8","Type":"ContainerStarted","Data":"8a7ab2efcf977cd5c0a29a4ec4c2b6dc6c460718cdb83ec7384c415d1ee67546"}
Oct 30 21:30:42.277991 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:42.277940    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj" podStartSLOduration=2.277926344 podStartE2EDuration="2.277926344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:42.276457112 +0000 UTC m=+4222.877259071" watchObservedRunningTime="2026-10-30 21:30:42.277926344 +0000 UTC m=+4222.878728304"
Oct 30 21:30:43.169351 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:43.169320    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f456893-663b-4a81-af12-6cca81861cc8" containerID="dbccb868dbc82ea425eb09e8fe3b6067855ec51ed2f033d3d9b92bf2c85a53fd" exitCode=0
Oct 30 21:30:43.169766 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:43.169400    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj" event={"ID":"2f456893-663b-4a81-af12-6cca81861cc8","Type":"ContainerDied","Data":"dbccb868dbc82ea425eb09e8fe3b6067855ec51ed2f033d3d9b92bf2c85a53fd"}
Oct 30 21:30:44.311788 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.311762    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:44.332874 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.332852    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca\") pod \"2f456893-663b-4a81-af12-6cca81861cc8\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") "
Oct 30 21:30:44.332991 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.332950    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca\") pod \"2f456893-663b-4a81-af12-6cca81861cc8\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") "
Oct 30 21:30:44.333039 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.332998    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9\" (UniqueName: \"kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9\") pod \"2f456893-663b-4a81-af12-6cca81861cc8\" (UID: \"2f456893-663b-4a81-af12-6cca81861cc8\") "
Oct 30 21:30:44.333233 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.333208    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca" pod "2f456893-663b-4a81-af12-6cca81861cc8" (UID: "2f456893-663b-4a81-af12-6cca81861cc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:44.335029 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.334998    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca" pod "2f456893-663b-4a81-af12-6cca81861cc8" (UID: "2f456893-663b-4a81-af12-6cca81861cc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:44.335135 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.335068    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9" pod "2f456893-663b-4a81-af12-6cca81861cc8" (UID: "2f456893-663b-4a81-af12-6cca81861cc8"). InnerVolumeSpecName "kube-api-access-6w8c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:44.433776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.433721    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w8c9\" (UniqueName: \"kubernetes.io/projected/2f456893-663b-4a81-af12-6cca81861cc8-kube-api-access-6w8c9\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:44.433776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.433755    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:44.433776 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:44.433764    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f456893-663b-4a81-af12-6cca81861cc8-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:45.180389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:45.180361    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj"
Oct 30 21:30:45.180590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:45.180399    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-79mxj" event={"ID":"2f456893-663b-4a81-af12-6cca81861cc8","Type":"ContainerDied","Data":"8a7ab2efcf977cd5c0a29a4ec4c2b6dc6c460718cdb83ec7384c415d1ee67546"}
Oct 30 21:30:45.180590 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:30:45.180470    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a7ab2efcf977cd5c0a29a4ec4c2b6dc6c460718cdb83ec7384c415d1ee67546"
Oct 30 21:31:01.563270 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563197    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"]
Oct 30 21:31:01.563631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563620    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2f456893-663b-4a81-af12-6cca81861cc8" containerName="revoke-keys"
Oct 30 21:31:01.563675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563631    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f456893-663b-4a81-af12-6cca81861cc8" containerName="revoke-keys"
Oct 30 21:31:01.563675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563653    5967 cpu_manager.go:414] "containerMap: removing container" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:31:01.563675 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563658    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:31:01.563795 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563719    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f456893-663b-4a81-af12-6cca81861cc8" containerName="revoke-keys"
Oct 30 21:31:01.563795 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.563732    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="186934a5-b0ef-49a7-8582-139757240d7e" containerName="cleanup"
Oct 30 21:31:01.566956 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.566937    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.623324 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.623301    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"]
Oct 30 21:31:01.670055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.670030    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:01.670055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.670031    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:01.670055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.670039    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8nnvq\""
Oct 30 21:31:01.670291 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.670137    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:01.734913 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.734884    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.735052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.734937    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72nh4\" (UniqueName: \"kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.735052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.735041    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.835970 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.835904    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.835970 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.835959    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.836163 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.836028    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72nh4\" (UniqueName: \"kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.836641 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.836623    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.838481 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.838465    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.892551 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:01.892525    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72nh4\" (UniqueName: \"kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4\") pod \"maas-api-revoke-sub-0165d7d2-vsz2g\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:01.978106 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:31:01.978076    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:31:02.184042 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:02.183968    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:02.552369 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:02.552320    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"]
Oct 30 21:31:02.554692 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:31:02.554658    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f68a5d3_0803_4422_a8eb_fc74479e2159.slice/crio-8a3b5dc7bcb36d4723c3374a1e5704910a3e2a450cbf3124e907e010cce8b913 WatchSource:0}: Error finding container 8a3b5dc7bcb36d4723c3374a1e5704910a3e2a450cbf3124e907e010cce8b913: Status 404 returned error can't find the container with id 8a3b5dc7bcb36d4723c3374a1e5704910a3e2a450cbf3124e907e010cce8b913
Oct 30 21:31:03.277407 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:03.277375    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g" event={"ID":"2f68a5d3-0803-4422-a8eb-fc74479e2159","Type":"ContainerStarted","Data":"85e77fc065f77d72f73a7fff0feec1fa736b218da78d3ff49a7fa454649e876e"}
Oct 30 21:31:03.277920 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:03.277424    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g" event={"ID":"2f68a5d3-0803-4422-a8eb-fc74479e2159","Type":"ContainerStarted","Data":"8a3b5dc7bcb36d4723c3374a1e5704910a3e2a450cbf3124e907e010cce8b913"}
Oct 30 21:31:03.392941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:03.392894    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g" podStartSLOduration=2.39287998 podStartE2EDuration="2.39287998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:03.390991916 +0000 UTC m=+4243.991793874" watchObservedRunningTime="2026-10-30 21:31:03.39287998 +0000 UTC m=+4243.993681939"
Oct 30 21:31:04.045362 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:04.045332    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"]
Oct 30 21:31:04.094827 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:04.094796    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5lwqx"]
Oct 30 21:31:04.284052 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:04.284016    5967 generic.go:356] "Generic (PLEG): container finished" podID="2f68a5d3-0803-4422-a8eb-fc74479e2159" containerID="85e77fc065f77d72f73a7fff0feec1fa736b218da78d3ff49a7fa454649e876e" exitCode=0
Oct 30 21:31:04.284457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:04.284107    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g" event={"ID":"2f68a5d3-0803-4422-a8eb-fc74479e2159","Type":"ContainerDied","Data":"85e77fc065f77d72f73a7fff0feec1fa736b218da78d3ff49a7fa454649e876e"}
Oct 30 21:31:05.439725 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.439702    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:05.541236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.541177    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8d5107-3cbf-4866-9813-87e4c341dac2" path="/var/lib/kubelet/pods/3f8d5107-3cbf-4866-9813-87e4c341dac2/volumes"
Oct 30 21:31:05.570581 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.570556    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4\" (UniqueName: \"kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4\") pod \"2f68a5d3-0803-4422-a8eb-fc74479e2159\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") "
Oct 30 21:31:05.570712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.570601    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca\") pod \"2f68a5d3-0803-4422-a8eb-fc74479e2159\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") "
Oct 30 21:31:05.570712 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.570670    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca\") pod \"2f68a5d3-0803-4422-a8eb-fc74479e2159\" (UID: \"2f68a5d3-0803-4422-a8eb-fc74479e2159\") "
Oct 30 21:31:05.571055 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.571032    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca" pod "2f68a5d3-0803-4422-a8eb-fc74479e2159" (UID: "2f68a5d3-0803-4422-a8eb-fc74479e2159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:05.572805 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.572783    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4" pod "2f68a5d3-0803-4422-a8eb-fc74479e2159" (UID: "2f68a5d3-0803-4422-a8eb-fc74479e2159"). InnerVolumeSpecName "kube-api-access-72nh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:05.572904 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.572807    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca" pod "2f68a5d3-0803-4422-a8eb-fc74479e2159" (UID: "2f68a5d3-0803-4422-a8eb-fc74479e2159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:05.672215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.672190    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72nh4\" (UniqueName: \"kubernetes.io/projected/2f68a5d3-0803-4422-a8eb-fc74479e2159-kube-api-access-72nh4\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:05.672215 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.672212    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:05.672361 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:05.672223    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f68a5d3-0803-4422-a8eb-fc74479e2159-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:06.295780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:06.295749    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g"
Oct 30 21:31:06.295780 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:06.295764    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vsz2g" event={"ID":"2f68a5d3-0803-4422-a8eb-fc74479e2159","Type":"ContainerDied","Data":"8a3b5dc7bcb36d4723c3374a1e5704910a3e2a450cbf3124e907e010cce8b913"}
Oct 30 21:31:06.296066 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:06.295832    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a3b5dc7bcb36d4723c3374a1e5704910a3e2a450cbf3124e907e010cce8b913"
Oct 30 21:31:13.273150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.273115    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"]
Oct 30 21:31:13.273609 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.273593    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2f68a5d3-0803-4422-a8eb-fc74479e2159" containerName="revoke-keys"
Oct 30 21:31:13.273662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.273612    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f68a5d3-0803-4422-a8eb-fc74479e2159" containerName="revoke-keys"
Oct 30 21:31:13.273699 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.273692    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f68a5d3-0803-4422-a8eb-fc74479e2159" containerName="revoke-keys"
Oct 30 21:31:13.278059 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.278044    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.344151 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.344112    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwzf6\" (UniqueName: \"kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.344343 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.344263    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.344450 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.344434    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.381577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.381547    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:13.381577 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.381572    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:13.381783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.381623    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:13.381783 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.381666    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8nnvq\""
Oct 30 21:31:13.383379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.383360    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"]
Oct 30 21:31:13.445695 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.445666    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.445869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.445791    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.445869 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.445825    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwzf6\" (UniqueName: \"kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.446354 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.446335    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.448257 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.448235    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.502252 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.502219    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwzf6\" (UniqueName: \"kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6\") pod \"maas-api-revoke-sub-6b19338f-fvkjs\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.590651 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.590591    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:13.960566 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:13.960486    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"]
Oct 30 21:31:13.963006 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:31:13.962984    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a00b236_df4a_4dea_bda7_b0080e0bbed6.slice/crio-ad956e769d259228d5d098107a2c3fb459d9366318e467a23111bcd940b2322d WatchSource:0}: Error finding container ad956e769d259228d5d098107a2c3fb459d9366318e467a23111bcd940b2322d: Status 404 returned error can't find the container with id ad956e769d259228d5d098107a2c3fb459d9366318e467a23111bcd940b2322d
Oct 30 21:31:14.343457 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:14.343425    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs" event={"ID":"2a00b236-df4a-4dea-bda7-b0080e0bbed6","Type":"ContainerStarted","Data":"172e0a4312a564feff6f84c81cbd3fcdc0b0220e88e7d5f423d38c5b2b43ef23"}
Oct 30 21:31:14.343834 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:14.343479    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs" event={"ID":"2a00b236-df4a-4dea-bda7-b0080e0bbed6","Type":"ContainerStarted","Data":"ad956e769d259228d5d098107a2c3fb459d9366318e467a23111bcd940b2322d"}
Oct 30 21:31:14.472886 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:14.472842    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs" podStartSLOduration=1.472827246 podStartE2EDuration="1.472827246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:14.47186595 +0000 UTC m=+4255.072667911" watchObservedRunningTime="2026-10-30 21:31:14.472827246 +0000 UTC m=+4255.073629207"
Oct 30 21:31:15.350425 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:15.350395    5967 generic.go:356] "Generic (PLEG): container finished" podID="2a00b236-df4a-4dea-bda7-b0080e0bbed6" containerID="172e0a4312a564feff6f84c81cbd3fcdc0b0220e88e7d5f423d38c5b2b43ef23" exitCode=0
Oct 30 21:31:15.350837 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:15.350458    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs" event={"ID":"2a00b236-df4a-4dea-bda7-b0080e0bbed6","Type":"ContainerDied","Data":"172e0a4312a564feff6f84c81cbd3fcdc0b0220e88e7d5f423d38c5b2b43ef23"}
Oct 30 21:31:16.489110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.489088    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:16.576798 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.576774    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca\") pod \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") "
Oct 30 21:31:16.576919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.576849    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca\") pod \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") "
Oct 30 21:31:16.576919 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.576879    5967 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6\" (UniqueName: \"kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6\") pod \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\" (UID: \"2a00b236-df4a-4dea-bda7-b0080e0bbed6\") "
Oct 30 21:31:16.577170 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.577143    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca" pod "2a00b236-df4a-4dea-bda7-b0080e0bbed6" (UID: "2a00b236-df4a-4dea-bda7-b0080e0bbed6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:16.577297 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.577250    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-openshift-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:16.578909 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.578887    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca" pod "2a00b236-df4a-4dea-bda7-b0080e0bbed6" (UID: "2a00b236-df4a-4dea-bda7-b0080e0bbed6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:16.579016 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.578958    5967 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6" pod "2a00b236-df4a-4dea-bda7-b0080e0bbed6" (UID: "2a00b236-df4a-4dea-bda7-b0080e0bbed6"). InnerVolumeSpecName "kube-api-access-rwzf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:16.678247 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.678181    5967 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a00b236-df4a-4dea-bda7-b0080e0bbed6-maas-api-xks-ca\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:16.678247 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:16.678205    5967 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwzf6\" (UniqueName: \"kubernetes.io/projected/2a00b236-df4a-4dea-bda7-b0080e0bbed6-kube-api-access-rwzf6\") on node \"ip-10-0-130-195.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:17.362308 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:17.362273    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs" event={"ID":"2a00b236-df4a-4dea-bda7-b0080e0bbed6","Type":"ContainerDied","Data":"ad956e769d259228d5d098107a2c3fb459d9366318e467a23111bcd940b2322d"}
Oct 30 21:31:17.362498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:17.362332    5967 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad956e769d259228d5d098107a2c3fb459d9366318e467a23111bcd940b2322d"
Oct 30 21:31:17.362498 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:17.362289    5967 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fvkjs"
Oct 30 21:31:31.979469 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:31:31.979436    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:31:33.044853 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:33.044779    5967 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"]
Oct 30 21:31:33.094138 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:33.094110    5967 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40c647-nrttr"]
Oct 30 21:31:33.540785 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:33.540729    5967 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ac9137-f50c-4c47-a638-f3d5b170bf5f" path="/var/lib/kubelet/pods/36ac9137-f50c-4c47-a638-f3d5b170bf5f/volumes"
Oct 30 21:31:36.321570 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:36.321541    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-85b76df47b-hf9k5_08b81ab9-ce89-45f4-8a12-37447542b8a4/manager/0.log"
Oct 30 21:31:38.060236 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:38.060203    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d9799fc-b67rj_9144ff77-360b-4134-b77d-e8e1f12005e9/manager/0.log"
Oct 30 21:31:38.634140 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:38.634116    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2fpxj_785df042-e446-406f-87d7-2325e753dbc0/server/0.log"
Oct 30 21:31:38.860319 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:38.860286    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6hgn6_a3057d5f-f9ff-4941-bca4-15efadba993e/manager/0.log"
Oct 30 21:31:40.639845 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:40.639814    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vsz2g_2f68a5d3-0803-4422-a8eb-fc74479e2159/revoke-keys/0.log"
Oct 30 21:31:40.854865 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:40.854822    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fvkjs_2a00b236-df4a-4dea-bda7-b0080e0bbed6/revoke-keys/0.log"
Oct 30 21:31:41.088656 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:41.088626    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-79mxj_2f456893-663b-4a81-af12-6cca81861cc8/revoke-keys/0.log"
Oct 30 21:31:43.577246 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:43.577218    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fnhlj_c4dbcd9d-614f-4f33-8280-2c4e75578ac5/manager/0.log"
Oct 30 21:31:43.851555 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:43.851473    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9n4jr_91d3c401-0f49-4263-a5f0-b6ac66470202/kuadrant-console-plugin/0.log"
Oct 30 21:31:46.114404 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:46.114374    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-557f979cb4-ctpdv_493b7e7c-228f-481c-9a88-aa6b798854e5/kube-auth-proxy/0.log"
Oct 30 21:31:47.366525 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:47.366491    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-d2bvn_426cdbf2-0251-4f31-aea0-b393f86ce297/istio-proxy/0.log"
Oct 30 21:31:47.686142 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:47.686061    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-f8t7j_f2cc343a-deb9-49a0-8103-d3a4eeb8b892/istio-proxy/0.log"
Oct 30 21:31:48.980623 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:48.980587    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-xbx2q_b0826915-086e-4d42-8ade-b517b6aa7925/istio-proxy/0.log"
Oct 30 21:31:49.487565 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:49.487535    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9bl76_4da87480-60d8-45a3-aeda-895907c9e229/payload-pre-processing/0.log"
Oct 30 21:31:49.718694 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:49.718660    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vtz7b_b20b8d66-af6a-4db2-a44a-f20e16cb834e/payload-processing/0.log"
Oct 30 21:31:50.853405 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:50.853373    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh_5d876e30-bfc0-4db6-a136-e13e456d3abc/main/0.log"
Oct 30 21:31:51.401059 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:51.401027    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-jsdnb_9bc25d7c-227f-4f78-9cfb-4cf5e3c69677/main/0.log"
Oct 30 21:31:52.630609 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:31:52.630578    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8_fd69661d-c5d4-446c-ad59-866e2e47adc7/main/0.log"
Oct 30 21:32:01.980305 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:32:01.980257    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:32:03.009051 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:03.009019    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-85b76df47b-hf9k5_08b81ab9-ce89-45f4-8a12-37447542b8a4/manager/0.log"
Oct 30 21:32:04.428136 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:04.428093    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d9799fc-b67rj_9144ff77-360b-4134-b77d-e8e1f12005e9/manager/0.log"
Oct 30 21:32:04.847202 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:04.847168    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2fpxj_785df042-e446-406f-87d7-2325e753dbc0/server/0.log"
Oct 30 21:32:05.066600 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:05.066573    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6hgn6_a3057d5f-f9ff-4941-bca4-15efadba993e/manager/0.log"
Oct 30 21:32:06.645826 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:06.645799    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vsz2g_2f68a5d3-0803-4422-a8eb-fc74479e2159/revoke-keys/0.log"
Oct 30 21:32:06.869021 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:06.868983    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fvkjs_2a00b236-df4a-4dea-bda7-b0080e0bbed6/revoke-keys/0.log"
Oct 30 21:32:07.098654 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:07.098618    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-79mxj_2f456893-663b-4a81-af12-6cca81861cc8/revoke-keys/0.log"
Oct 30 21:32:09.258510 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:09.258478    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fnhlj_c4dbcd9d-614f-4f33-8280-2c4e75578ac5/manager/0.log"
Oct 30 21:32:09.527560 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:09.527480    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9n4jr_91d3c401-0f49-4263-a5f0-b6ac66470202/kuadrant-console-plugin/0.log"
Oct 30 21:32:11.539420 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:11.539388    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-557f979cb4-ctpdv_493b7e7c-228f-481c-9a88-aa6b798854e5/kube-auth-proxy/0.log"
Oct 30 21:32:12.702495 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:12.702469    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-d2bvn_426cdbf2-0251-4f31-aea0-b393f86ce297/istio-proxy/0.log"
Oct 30 21:32:13.030365 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:13.030330    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-f8t7j_f2cc343a-deb9-49a0-8103-d3a4eeb8b892/istio-proxy/0.log"
Oct 30 21:32:14.297657 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:14.297626    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-xbx2q_b0826915-086e-4d42-8ade-b517b6aa7925/istio-proxy/0.log"
Oct 30 21:32:14.800345 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:14.800305    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9bl76_4da87480-60d8-45a3-aeda-895907c9e229/payload-pre-processing/0.log"
Oct 30 21:32:15.024208 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:15.024177    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vtz7b_b20b8d66-af6a-4db2-a44a-f20e16cb834e/payload-processing/0.log"
Oct 30 21:32:15.911602 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:15.911563    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mdggh_5d876e30-bfc0-4db6-a136-e13e456d3abc/main/0.log"
Oct 30 21:32:16.428366 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:16.428324    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-jsdnb_9bc25d7c-227f-4f78-9cfb-4cf5e3c69677/main/0.log"
Oct 30 21:32:17.609035 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:17.609002    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jmwk8_fd69661d-c5d4-446c-ad59-866e2e47adc7/main/0.log"
Oct 30 21:32:23.398082 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:23.398048    5967 scope.go:122] "RemoveContainer" containerID="5a3ccd06436886084e322ba8e3acc1e3996a079330acaaac3509faf29b1cf16c"
Oct 30 21:32:23.407501 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:23.407479    5967 scope.go:122] "RemoveContainer" containerID="dee43ce66a424f7d0ed1bf3465ccaf16a6e568084a9f900fefb217896bd393a4"
Oct 30 21:32:31.981981 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:32:31.981885    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:32:46.228419 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:46.228380    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d9799fc-b67rj_9144ff77-360b-4134-b77d-e8e1f12005e9/manager/0.log"
Oct 30 21:32:46.752794 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:46.752731    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d9799fc-b67rj_9144ff77-360b-4134-b77d-e8e1f12005e9/manager/0.log"
Oct 30 21:32:47.547138 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:47.547107    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vtz7b_b20b8d66-af6a-4db2-a44a-f20e16cb834e/payload-processing/0.log"
Oct 30 21:32:47.760123 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:32:47.760099    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vtz7b_b20b8d66-af6a-4db2-a44a-f20e16cb834e/payload-processing/0.log"
Oct 30 21:33:01.980965 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:33:01.980929    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:33:23.432360 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:23.432327    5967 scope.go:122] "RemoveContainer" containerID="f110dd20186f20bd5eb65d211263d2f07a15acf6d11b5b755b01431a1a938511"
Oct 30 21:33:24.795299 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:24.795263    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d9799fc-b67rj_9144ff77-360b-4134-b77d-e8e1f12005e9/manager/0.log"
Oct 30 21:33:25.377314 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:25.377265    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d9799fc-b67rj_9144ff77-360b-4134-b77d-e8e1f12005e9/manager/0.log"
Oct 30 21:33:26.187179 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.187152    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vtz7b_b20b8d66-af6a-4db2-a44a-f20e16cb834e/payload-processing/0.log"
Oct 30 21:33:26.410582 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.410552    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vtz7b_b20b8d66-af6a-4db2-a44a-f20e16cb834e/payload-processing/0.log"
Oct 30 21:33:26.751421 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.751384    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hsvt5/must-gather-2r8ft"]
Oct 30 21:33:26.752455 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.752426    5967 cpu_manager.go:414] "containerMap: removing container" podUID="2a00b236-df4a-4dea-bda7-b0080e0bbed6" containerName="revoke-keys"
Oct 30 21:33:26.752649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.752632    5967 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a00b236-df4a-4dea-bda7-b0080e0bbed6" containerName="revoke-keys"
Oct 30 21:33:26.753120 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.753100    5967 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a00b236-df4a-4dea-bda7-b0080e0bbed6" containerName="revoke-keys"
Oct 30 21:33:26.761385 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.761363    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:26.813110 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.813081    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsvt5\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.837154 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.837117    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57b2fe7c-7ce5-4dcd-a06f-b66524c5c658-must-gather-output\") pod \"must-gather-2r8ft\" (UID: \"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658\") " pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:26.837356 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.837243    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnm6q\" (UniqueName: \"kubernetes.io/projected/57b2fe7c-7ce5-4dcd-a06f-b66524c5c658-kube-api-access-lnm6q\") pod \"must-gather-2r8ft\" (UID: \"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658\") " pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:26.862514 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.862479    5967 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsvt5\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.862514 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.862513    5967 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hsvt5\"/\"default-dockercfg-gxfgx\""
Oct 30 21:33:26.864508 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.864487    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/must-gather-2r8ft"]
Oct 30 21:33:26.938685 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.938650    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnm6q\" (UniqueName: \"kubernetes.io/projected/57b2fe7c-7ce5-4dcd-a06f-b66524c5c658-kube-api-access-lnm6q\") pod \"must-gather-2r8ft\" (UID: \"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658\") " pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:26.938914 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.938749    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57b2fe7c-7ce5-4dcd-a06f-b66524c5c658-must-gather-output\") pod \"must-gather-2r8ft\" (UID: \"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658\") " pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:26.939122 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:26.939103    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57b2fe7c-7ce5-4dcd-a06f-b66524c5c658-must-gather-output\") pod \"must-gather-2r8ft\" (UID: \"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658\") " pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:27.003149 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:27.003071    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnm6q\" (UniqueName: \"kubernetes.io/projected/57b2fe7c-7ce5-4dcd-a06f-b66524c5c658-kube-api-access-lnm6q\") pod \"must-gather-2r8ft\" (UID: \"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658\") " pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:27.076072 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:27.076042    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/must-gather-2r8ft"
Oct 30 21:33:27.449150 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:27.449115    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/must-gather-2r8ft"]
Oct 30 21:33:27.451865 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:33:27.451829    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57b2fe7c_7ce5_4dcd_a06f_b66524c5c658.slice/crio-5d3a784d1afbb8f5a0ead1d3ec8a9f4e14953908fb6417080a8bcc62f3d7f4b9 WatchSource:0}: Error finding container 5d3a784d1afbb8f5a0ead1d3ec8a9f4e14953908fb6417080a8bcc62f3d7f4b9: Status 404 returned error can't find the container with id 5d3a784d1afbb8f5a0ead1d3ec8a9f4e14953908fb6417080a8bcc62f3d7f4b9
Oct 30 21:33:27.453855 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:27.453835    5967 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:33:28.080662 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:28.080621    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/must-gather-2r8ft" event={"ID":"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658","Type":"ContainerStarted","Data":"5d3a784d1afbb8f5a0ead1d3ec8a9f4e14953908fb6417080a8bcc62f3d7f4b9"}
Oct 30 21:33:29.088573 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:29.088539    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/must-gather-2r8ft" event={"ID":"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658","Type":"ContainerStarted","Data":"c7d727352dffb291bcb4d963e5a6c4e7b019bedd044a2b280fcda3e5859cd66c"}
Oct 30 21:33:29.089015 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:29.088585    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/must-gather-2r8ft" event={"ID":"57b2fe7c-7ce5-4dcd-a06f-b66524c5c658","Type":"ContainerStarted","Data":"32d11daf1499716aa8c16d758532df1de64cd6c9393fc701c7378899bddab25e"}
Oct 30 21:33:29.203077 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:29.202971    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hsvt5/must-gather-2r8ft" podStartSLOduration=1.973145266 podStartE2EDuration="3.202954616s" totalImagesPullingTime="1.22980935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:29.201609582 +0000 UTC m=+4389.802411542" watchObservedRunningTime="2026-10-30 21:33:29.202954616 +0000 UTC m=+4389.803756576"
Oct 30 21:33:31.980830 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:33:31.980783    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:33:36.244925 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:36.244893    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f84bz_f489b9e7-d156-4369-9afc-b38d675d1a31/global-pull-secret-syncer/0.log"
Oct 30 21:33:39.879559 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:39.879485    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pxcqk_4bded661-99cb-4a66-a36b-4e0a446321ef/konnectivity-agent/0.log"
Oct 30 21:33:40.977578 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:40.977538    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-195.us-west-1.compute.internal_066c09478c1923ae07faa25b9022b5b3/haproxy/0.log"
Oct 30 21:33:45.870726 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.870675    5967 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"]
Oct 30 21:33:45.879514 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.879488    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:45.933114 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.933076    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"]
Oct 30 21:33:45.942568 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.942527    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-sys\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:45.942797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.942638    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-podres\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:45.942797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.942668    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-lib-modules\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:45.942797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.942697    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzsd\" (UniqueName: \"kubernetes.io/projected/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-kube-api-access-bnzsd\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:45.942797 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:45.942792    5967 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-proc\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045643 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045605    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-sys\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045688    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-podres\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045705    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-lib-modules\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045723    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzsd\" (UniqueName: \"kubernetes.io/projected/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-kube-api-access-bnzsd\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045726    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-sys\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045822    5967 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-proc\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045845    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-podres\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045864    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-lib-modules\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.045903 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.045872    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-proc\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.104972 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.104938    5967 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzsd\" (UniqueName: \"kubernetes.io/projected/d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2-kube-api-access-bnzsd\") pod \"perf-node-gather-daemonset-vm77r\" (UID: \"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.137666 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.137569    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fnhlj_c4dbcd9d-614f-4f33-8280-2c4e75578ac5/manager/0.log"
Oct 30 21:33:46.193267 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.193229    5967 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:46.555412 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.555380    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9n4jr_91d3c401-0f49-4263-a5f0-b6ac66470202/kuadrant-console-plugin/0.log"
Oct 30 21:33:46.574941 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:46.574849    5967 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"]
Oct 30 21:33:46.577721 ip-10-0-130-195 kubenswrapper[5967]: W1030 21:33:46.577690    5967 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4f5fc68_d838_4e6b_9e20_ee4c333b8ec2.slice/crio-2def6f07d4176e721347d589eed333fbe98791116aba3d9f321e3c5ffa807538 WatchSource:0}: Error finding container 2def6f07d4176e721347d589eed333fbe98791116aba3d9f321e3c5ffa807538: Status 404 returned error can't find the container with id 2def6f07d4176e721347d589eed333fbe98791116aba3d9f321e3c5ffa807538
Oct 30 21:33:47.229730 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:47.229688    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r" event={"ID":"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2","Type":"ContainerStarted","Data":"8c5833ed046221c8edba2982f475a25c3f31475b9d1ae8365b28327cd726ee55"}
Oct 30 21:33:47.230313 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:47.229783    5967 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r" event={"ID":"d4f5fc68-d838-4e6b-9e20-ee4c333b8ec2","Type":"ContainerStarted","Data":"2def6f07d4176e721347d589eed333fbe98791116aba3d9f321e3c5ffa807538"}
Oct 30 21:33:47.230313 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:47.230012    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:47.347649 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:47.347580    5967 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r" podStartSLOduration=2.347558099 podStartE2EDuration="2.347558099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:47.345281872 +0000 UTC m=+4407.946083834" watchObservedRunningTime="2026-10-30 21:33:47.347558099 +0000 UTC m=+4407.948360061"
Oct 30 21:33:50.471786 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:50.471729    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/alertmanager/0.log"
Oct 30 21:33:50.890144 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:50.890065    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/config-reloader/0.log"
Oct 30 21:33:51.307825 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:51.307792    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/kube-rbac-proxy-web/0.log"
Oct 30 21:33:51.725540 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:51.725507    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/kube-rbac-proxy/0.log"
Oct 30 21:33:52.141174 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:52.141094    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/kube-rbac-proxy-metric/0.log"
Oct 30 21:33:52.557809 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:52.557783    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/prom-label-proxy/0.log"
Oct 30 21:33:52.975000 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:52.974974    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b0811cb0-c2cc-4560-a61e-1f2077870d7f/init-config-reloader/0.log"
Oct 30 21:33:53.253208 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:53.253128    5967 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-vm77r"
Oct 30 21:33:53.917480 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:53.917438    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9v82b_a968a685-34c9-4b22-9bd6-0963e6f2dc5a/kube-state-metrics/0.log"
Oct 30 21:33:54.371670 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:54.371639    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9v82b_a968a685-34c9-4b22-9bd6-0963e6f2dc5a/kube-rbac-proxy-main/0.log"
Oct 30 21:33:54.791631 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:54.791603    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9v82b_a968a685-34c9-4b22-9bd6-0963e6f2dc5a/kube-rbac-proxy-self/0.log"
Oct 30 21:33:55.636700 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:55.636665    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-lwh2w_ff0b92cc-28ad-435b-bbcb-161bb00e2c42/monitoring-plugin/0.log"
Oct 30 21:33:58.578954 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:58.578871    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9bdp_81ab7c91-f29a-4204-b268-5407d7129914/node-exporter/0.log"
Oct 30 21:33:58.995264 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:58.995232    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9bdp_81ab7c91-f29a-4204-b268-5407d7129914/kube-rbac-proxy/0.log"
Oct 30 21:33:59.414362 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:59.414281    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9bdp_81ab7c91-f29a-4204-b268-5407d7129914/init-textfile/0.log"
Oct 30 21:33:59.834379 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:33:59.834345    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7rhr6_ec59824c-8bdf-4099-9124-26a29ae8e876/kube-rbac-proxy-main/0.log"
Oct 30 21:34:00.253108 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:00.253033    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7rhr6_ec59824c-8bdf-4099-9124-26a29ae8e876/kube-rbac-proxy-self/0.log"
Oct 30 21:34:00.668621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:00.668538    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7rhr6_ec59824c-8bdf-4099-9124-26a29ae8e876/openshift-state-metrics/0.log"
Oct 30 21:34:01.097651 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:01.097616    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/prometheus/0.log"
Oct 30 21:34:01.513213 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:01.513174    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/config-reloader/0.log"
Oct 30 21:34:01.930762 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:01.930666    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/thanos-sidecar/0.log"
Oct 30 21:34:01.985966 ip-10-0-130-195 kubenswrapper[5967]: E1030 21:34:01.985932    5967 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9zjw5" containerName="extractor"
Oct 30 21:34:02.348621 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:02.348594    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/kube-rbac-proxy-web/0.log"
Oct 30 21:34:02.768386 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:02.768356    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/kube-rbac-proxy/0.log"
Oct 30 21:34:03.188432 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:03.188351    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/kube-rbac-proxy-thanos/0.log"
Oct 30 21:34:03.606118 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:03.606084    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_05d83498-1a98-4352-81f1-2edb48492715/init-config-reloader/0.log"
Oct 30 21:34:04.029478 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:04.029444    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-f424d_79d81e86-e9cb-4bf1-9286-74af62e818ee/prometheus-operator/0.log"
Oct 30 21:34:04.447344 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:04.447270    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-f424d_79d81e86-e9cb-4bf1-9286-74af62e818ee/kube-rbac-proxy/0.log"
Oct 30 21:34:04.866677 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:04.866645    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-nnfbg_de295ab7-6640-4744-826b-e780a3d7e597/prometheus-operator-admission-webhook/0.log"
Oct 30 21:34:06.546389 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:06.546358    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c64476cfd-sv4hq_f575738a-d51b-460e-9ad5-e2aeb948f4bc/thanos-query/0.log"
Oct 30 21:34:06.962580 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:06.962485    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c64476cfd-sv4hq_f575738a-d51b-460e-9ad5-e2aeb948f4bc/kube-rbac-proxy-web/0.log"
Oct 30 21:34:07.378100 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:07.378075    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c64476cfd-sv4hq_f575738a-d51b-460e-9ad5-e2aeb948f4bc/kube-rbac-proxy/0.log"
Oct 30 21:34:07.795203 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:07.795161    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c64476cfd-sv4hq_f575738a-d51b-460e-9ad5-e2aeb948f4bc/prom-label-proxy/0.log"
Oct 30 21:34:08.213821 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:08.213441    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c64476cfd-sv4hq_f575738a-d51b-460e-9ad5-e2aeb948f4bc/kube-rbac-proxy-rules/0.log"
Oct 30 21:34:08.629250 ip-10-0-130-195 kubenswrapper[5967]: I1030 21:34:08.629218    5967 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c64476cfd-sv4hq_f575738a-d51b-460e-9ad5-e2aeb948f4bc/kube-rbac-proxy-metrics/0.log"